Welcome to the FTS Forum - Read the introduction here

Avatar
Please consider registering
Guest
Search
Forum Scope


Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Feed
Topic Rating: +5 Topic Rating: +5 (5 votes) 
sp_TopicIcon
How Do You Know Your 'Profitable' System Won't Regress Back To Zero?
Avatar
jamesh94
Beginner
Members
Forum Posts: 4
Member Since:
June 16, 2020
sp_UserOfflineSmall Offline
1
October 26, 2020 - 3:34 pm
sp_Permalink sp_Print
0

I’ve been a member of FDD for a few months now and have purchased quite a lot of historic data so I’ve got quite a lot of data to sort through when finding systems.

I’ve found a few systems and niches that seem to be consistently profitable, but then I’ve also found a few that do well for a couple of seasons and then regress back to zero.

So how do you know that your ‘profitable’ system won’t regress back to zero?

I guess this is the million-dollar question and you can never really ‘know’ until you start placing bets, but I’d imagine a few of you have certain stress-tests that your systems have to pass before you start using them. Would anyone be willing to share what their tests are?

For me, I use football-data.co.uk ‘s P-value calculator (available here: https://www.football-data.co.u…..lator.xlsx ) and insist on a 1-in-x probability of more than 100. This essentially tells you that there is a less than 1/100 chance of you getting these results just by being lucky.

So what else do people do before they start putting money down on a system? Looking forward to hearing from people on here.

James

Avatar
ikeee64
Beginner
Members

KickStarter 2020
Forum Posts: 55
Member Since:
February 8, 2018
sp_UserOfflineSmall Offline
2
October 27, 2020 - 10:05 am
sp_Permalink sp_Print
0

Hi, I was on Ian’s Kickstarter course and we was told to test any systems we running every 100 bets to see how the edge is preforming, gather for us this is the indicator we look for to see if that system edge is eroding, also like you using historical data these systems was back tested for the last 3 seasons and just keep monitoring going forward.

The following users say thank you to ikeee64 for this useful post:

jamesh94
Avatar
spanias7
Beginner
Members
Forum Posts: 2
Member Since:
August 7, 2019
sp_UserOfflineSmall Offline
3
October 27, 2020 - 10:50 am
sp_Permalink sp_Print
0

How would you determine an edge eroding vs a drawdown?

You could have the last 100 bets which contain a drawdown and pick up again after the 100 bets? 

Especially if a dog based system (odds 3-4 and above) 

Interested to hear people thoughts 

The following users say thank you to spanias7 for this useful post:

jamesh94
Avatar
jamesh94
Beginner
Members
Forum Posts: 4
Member Since:
June 16, 2020
sp_UserOfflineSmall Offline
4
October 27, 2020 - 4:05 pm
sp_Permalink sp_Print
0

ikeee64 said
Hi, I was on Ian’s Kickstarter course and we was told to test any systems we running every 100 bets to see how the edge is preforming, gather for us this is the indicator we look for to see if that system edge is eroding, also like you using historical data these systems was back tested for the last 3 seasons and just keep monitoring going forward.

  

Cheers for the reply. Is 100 bets really enough of a sample to determine whether you’re losing an edge though?

e.g. say you had an edge with a system that won 55% of the time at odds of 2.0, but over the last 100 bets you only won 48%, would you consider your edge to have eroded? For me, that would just be variance and I’d expect to have losing periods over 100 bets.

Avatar
Lord Dio
Ultimate
Members
Forum Posts: 131
Member Since:
October 23, 2019
sp_UserOfflineSmall Offline
5
October 28, 2020 - 8:39 pm
sp_Permalink sp_Print
+2

jamesh94 said
I’ve been a member of FDD for a few months now and have purchased quite a lot of historic data so I’ve got quite a lot of data to sort through when finding systems.

I’ve found a few systems and niches that seem to be consistently profitable, but then I’ve also found a few that do well for a couple of seasons and then regress back to zero.

So how do you know that your ‘profitable’ system won’t regress back to zero?

I guess this is the million-dollar question and you can never really ‘know’ until you start placing bets, but I’d imagine a few of you have certain stress-tests that your systems have to pass before you start using them. Would anyone be willing to share what their tests are?

For me, I use football-data.co.uk ‘s P-value calculator (available here: https://www.football-data.co.u…..lator.xlsx ) and insist on a 1-in-x probability of more than 100. This essentially tells you that there is a less than 1/100 chance of you getting these results just by being lucky.

So what else do people do before they start putting money down on a system? Looking forward to hearing from people on here.

James

  

There are a few errors with your thought process above. 

You can only perform a T-test on bets you have actually placed. If you data-mine/back test a system, your “MO” is to find an edge and force a profit to appear by tweaking odds, scorelines, stats etc. The very nature of it is random and you will only ever run your test on a method that appears profitable but these results on paper are not replicated 99 times out of 100 when you come to bet for real. 

I would not be focussing on seasonal trends and looking which were “profitable” or not, the key would be to get real money bets down based on this system, get a sample of hundreds and THEN run the testing. 

Remember a P score will change as your input variables change. You might have 0.00001 after 100 bets but then 0.05 after 300 bets etc so it isn’t the be all and end all.

Ultimately, the end result you want is profit. If you can test a system with real money be it £2 or £200 stakes and it is profitable after weeks/months/years, you will probably carry on regardless of what the statistical tests tell you. Just some food for thought there. 

The following users say thank you to Lord Dio for this useful post:

Neil, becker
Avatar
Beeble
Beginner
Members
Forum Posts: 50
Member Since:
July 31, 2019
sp_UserOfflineSmall Offline
6
October 29, 2020 - 8:41 am
sp_Permalink sp_Print sp_EditHistory
+1

One of the problems with mining data is over-fitting, so what looks like a set of promising filters actually has no predictive power at all, and when you come to placing bets in real time the system either flat-lines or loses money.

There are a couple of things you can do to help:

  1. Paper trade (or place very small bets) for a reasonable number of bets and see if the trend continues as expected
  2. When mining the data only mine from a subset of the historical data (say 80%) and leave aside the rest for out of sample testing.  If the results in the out of sample data match closely with the in sample data then you can be more confident that you’ve got a robust system.

A book that I’ve found very useful for this sort of thing is ‘The Evaluation and Optimization of Trading Strategies’ by Robert Parlo.  It focuses on trading stocks and shares, but the principals discussed in it are applicable to sports betting and trading too.

As an example of how things won’t necessarily follow past performance. Two pictures of the P/L of a couple of systems I’m paper trading:

The blue lines are from back testing (using out of sample testing) and the orange lines are paper trades.  As you can see, one’s flat-lining and one’s going south, so even with out of sample testing there’s no guarantee of success.

The following users say thank you to Beeble for this useful post:

Dixon, Neil, chancandan, Lord Dio, becker
Avatar
spanias7
Beginner
Members
Forum Posts: 2
Member Since:
August 7, 2019
sp_UserOfflineSmall Offline
7
October 29, 2020 - 10:33 am
sp_Permalink sp_Print
0

That’s a really helpful post there beeble, 

I’ve had this happen to atleast 3 of my systems. 1 of which literally fell off a cliff. 

So with your 2x systems above that’s flat lined and dropped will you continue to test or will you drop them? How long would you continue to monitor for. 

Also say if they continued an upward trend and you began to bet with full stakes what’s to say if wont drop then? 

Hope you dont mind the barrage of questions lol 

Avatar
Beeble
Beginner
Members
Forum Posts: 50
Member Since:
July 31, 2019
sp_UserOfflineSmall Offline
8
October 29, 2020 - 10:53 am
sp_Permalink sp_Print
+2

I’ll carry on monitoring them, probably for another 100-200 or so trades and see where they are.  I’d then compare things like the strike rates, average wins, average losses, the standard deviation of the wins and losses to the back fitted data and see if they’re within what I’d expect.

If they had an upward trend I’d bet them but at small stakes to start with. If they continued then I’d ramp the stakes up.

There’s nothing to say they won’t eventually drop off, but if you monitor them at regular intervals then you can make a judgement as to whether it’s just a bad run or the edge has gone.

The book I mentioned suggests the metrics to look at when assessing how a strategy is doing. 

The following users say thank you to Beeble for this useful post:

Dixon, chancandan, Lord Dio
Avatar
Lord Dio
Ultimate
Members
Forum Posts: 131
Member Since:
October 23, 2019
sp_UserOfflineSmall Offline
9
October 29, 2020 - 12:11 pm
sp_Permalink sp_Print
0

Beeble said
One of the problems with mining data is over-fitting, so what looks like a set of promising filters actually has no predictive power at all, and when you come to placing bets in real time the system either flat-lines or loses money.

There are a couple of things you can do to help:

    1. Paper trade (or place very small bets) for a reasonable number of bets and see if the trend continues as expected
    1. When mining the data only mine from a subset of the historical data (say 80%) and leave aside the rest for out of sample testing.  If the results in the out of sample data match closely with the in sample data then you can be more confident that you’ve got a robust system.

A book that I’ve found very useful for this sort of thing is ‘The Evaluation and Optimization of Trading Strategies’ by Robert Parlo.  It focuses on trading stocks and shares, but the principals discussed in it are applicable to sports betting and trading too.

As an example of how things won’t necessarily follow past performance. Two pictures of the P/L of a couple of systems I’m paper trading:

The blue lines are from back testing (using out of sample testing) and the orange lines are paper trades.  As you can see, one’s flat-lining and one’s going south, so even with out of sample testing there’s no guarantee of success.

  

That’s a really helpful way of visualising it and showing what I was thinking. 

I guess another question to ask is “is there logic in the filtering/selection process” in terms of WHY the edge exists. I tend to find that the odd back tested method that does continue for work for real going forwards has the most logic in its process and is less random. 

Avatar
jamesh94
Beginner
Members
Forum Posts: 4
Member Since:
June 16, 2020
sp_UserOfflineSmall Offline
10
October 29, 2020 - 3:50 pm
sp_Permalink sp_Print
0

Cheers for the replies Dio and Beeble, certainly a lot to think about and work on.

Beeble – I’ve found a PDF of the Parlo book you mentioned so I will start to take a look through this now.

 

PDF for anyone interested: https://www.arabictrader.com/c…..69349D.pdf

Book seems to cost around £60 from Amazon so that will hopefully save others a few quid.

The following users say thank you to jamesh94 for this useful post:

johnpetrie86
Avatar
Lord Dio
Ultimate
Members
Forum Posts: 131
Member Since:
October 23, 2019
sp_UserOfflineSmall Offline
11
October 29, 2020 - 7:59 pm
sp_Permalink sp_Print
0

jamesh94 said
Cheers for the replies Dio and Beeble, certainly a lot to think about and work on.

Beeble – I’ve found a PDF of the Parlo book you mentioned so I will start to take a look through this now.

 

PDF for anyone interested: https://www.arabictrader.com/c…..69349D.pdf

Book seems to cost around £60 from Amazon so that will hopefully save others a few quid.

  

Just seen that, crazy price but recall some of my other trading books were similar price. 

Avatar
gurds1991
Beginner
Members
Forum Posts: 54
Member Since:
April 7, 2015
sp_UserOfflineSmall Offline
12
October 29, 2020 - 10:28 pm
sp_Permalink sp_Print
0

Is a good point, there is always risk though. Gambling is one of the highest risk activities, lot riskier than people involved in real estate investing. You just have to learn from the experiences as you go along, good stuff and bad stuff. 

When we merge our desire with our faith we can take action from a place of peace rather than control

Forum Timezone: Europe/London
Most Users Ever Online: 92
Currently Online: walton6, yorkiec, Johnno, blueboy, drew_c, hansbager
Guest(s) 3
Currently Browsing this Page:
1 Guest(s)
Top Posters:
Lord Dio: 131
leewillo75: 104
@capitalchrisw: 100
Neil: 77
des911: 74
Callum: 68
jamiep: 60
ikeee64: 55
gurds1991: 54
jb8426: 52
Newest Members:
tnelywqxrhzb
matthewlic
invillmum
ljqvasye
alrmomzg
ebqdaxaqjmiwr
lesliegow
lewismcd
foiebqkf
davidssasw
Forum Stats:
Groups: 5
Forums: 24
Topics: 359
Posts: 2651

 

Member Stats:
Guest Posters: 3
Members: 5083
Moderators: 2
Admins: 2
Administrators: Ian, Geeky Boy
Moderators: ArtTest, davefts