A Back Testing / Live Testing Dilemma


Ok guys. Here's the dilemma I'm having.

I've been live trading a single methodology/strategy since the last week of August and it has consistently made profits. By consistent I mean the profits were all around the same at the end of the month, with the exception of November which was double, and the drawdowns never exceeded my original backtesting forecasts. Basically everything was what it shouldve been according to my original backtesting.

Here's the issue though. I recently backtested again to see if I had improved because I felt I had, and out of the 3 or 4 months I backtested, I did consistently worse. These months I backtested were all during the high volatile times although one was during the summer. These months would've been in my original backtesting and I have those numbers still saved on my computer, which is why I could say my new backtesting was consistently worse.

So on one hand, real results do speak for themself and I haven't been around long enough to know if doing well for 4 months is long enough to be legit or just a fluke. On the other hand, it's kind of worrying that the original backtesting that started me on this method no longer seems to be valid for whatever reason.

What should I think and do about all this? I can feel the fear and doubt creeping in and I want to get this sorted out before the New Year.
Originally posted by feng456

...I recently backtested again to see if I had improved because I felt I had, and out of the 3 or 4 months I backtested, I did consistently worse...

I don't fully understand what you're saying... Are you comparing your actual results to the back tested results? i.e. You should have done better over that period?
i recorded my actual results from sept-dec.

i recorded my orginal backtesting results from jan-aug.

i found that when i went back and backtested again with the newly acquired 4 months of real live trading experience, i fared worse in some months during that jan - aug period than when i originally backtested.
I struggle to understand how the back tested results from January to August would change unless you changed the rules for your trading system?

Typically after you've been trading your system live you run a back test against the data to see where you should have entered and exited and compare your hypothetical results to your actual results and see why they differ. They will almost always be different because there will be trading days and times that you miss which will have a positive or negative influence on your results.
My system is not fully mechanical that's why. I built a discretionary part into it.

So is 4 months of live trading worth anything or could easily by a fluke?
Four months is an excellent start but could also be a fluke. The ultimate test of any system is when the system has experienced "all" market conditions.

Obviously it's impossible to anticipate "all" conditions because there are market conditions yet to come that no one has ever seen, that's the nature of the beast.

However, since 2000 we've seen two good market corrects along with a couple of bull markets. If you can back test your system through the last decade and it works then I would put good money behind it.
Given enough time, all trading systems revert to the mean.

It sounds like your system may be demonstrating sensitivity to it's starting point. For example, starting the historical test on the first trading day of the month produces significantly different results than starting on the 10th day of the month. I would generate a set of test results starting on each trading day of the month and going out for a few months, then do an analysis of variance on those results and see what your sensitivity (starting point bias) is. How sensitive is the system performance to the starting day of the test run ? The less sensitive the system is to it's starting date the more robust it is and more reliable the overall performance results.

Another thing to try is to break your backtest into smaller windows of time covering other periods of the market history that have similar volatility characteristics and see if that set of tests produces performance results with a lower overall variance. In this test the variance or sensitivity should narrow down if the system is robust with respect to the specific type of market condition chosen (level of volatility). From there you can run another subset of tests on a different volatility level and determine the overall sensitivity of the system to volatility. How well does the system adapt to a significant change in volatility ?