The correlation between the S&P 500 over the past four years (black and white candles in the chart below) and the four years leading up to the 1937 top (blue candles) is roughly 94%.
Please tell me you didn't just correlate two price series :/
Prices aren't stationary, which means (essentially) that they move around and compound on themselves. Correlation assumes that the two time series being correlated are stationary. If you want to correlate two price series, you do so using returns (e.g. +1%, -2%, ...), which are a stationary series. If you run a correlation on price series, you'll get silly high values like 94%.
My understanding is that using the first difference of the log is just an approximation of % change that is in common usage primarily because it was computationally faster and because it's closed form is smooth. I could be wrong though. Is there some advantage to doing that that i'm unaware of?
In unsophisticated terms, I think it's basically that you can probably play with nearly any two time series graphs and then manipulate the data such that you find correlations.
Of course, that doesn't mean there actually is any correlation.
I agree that's a bigger fundamental issue, but I still think that if you take two different arbitrary windows from two time series of pricing data and then adjust the scaling of one to "match" the other, one can "find" many of these "correlations", even within the same pair of time series.
Ya, you're right. That is an issue as well, though I think it's a second order one here. His correlation likely disappears if he does it on the returns. But spurious correlations, even done mathematically correctly, abound.
Comments
Please tell me you didn't just correlate two price series :/
Why is that a bad thing?
Prices aren't stationary, which means (essentially) that they move around and compound on themselves. Correlation assumes that the two time series being correlated are stationary. If you want to correlate two price series, you do so using returns (e.g. +1%, -2%, ...), which are a stationary series. If you run a correlation on price series, you'll get silly high values like 94%.
I'd go further and suggest using changes in the logs of the price over time.
My understanding is that using the first difference of the log is just an approximation of % change that is in common usage primarily because it was computationally faster and because it's closed form is smooth. I could be wrong though. Is there some advantage to doing that that i'm unaware of?
In unsophisticated terms, I think it's basically that you can probably play with nearly any two time series graphs and then manipulate the data such that you find correlations.
Of course, that doesn't mean there actually is any correlation.
No, it's a specific issue with price series. The correct way to do it is to correlate the return series.
I agree that's a bigger fundamental issue, but I still think that if you take two different arbitrary windows from two time series of pricing data and then adjust the scaling of one to "match" the other, one can "find" many of these "correlations", even within the same pair of time series.
Ya, you're right. That is an issue as well, though I think it's a second order one here. His correlation likely disappears if he does it on the returns. But spurious correlations, even done mathematically correctly, abound.