If you have a univariate series, just single values following each other -
[5, 3, 3, 2, 2, 2, 1, …]
What is the next number? Well let’s start with the search space - what is the possible range of the next number? Assuming unsigned 32bit integers (for explanation simplicity) it’s 0-(2^32-1)
So are all of those possible outputs equally likely? The next number could be 1, or it could be 345,654,543 … are those outputs equally likely?
Even though we know nothing about this sequence, most time series don’t make enormous random jumps, so no, they are not equally likely, 1 is the more likely of the two we discussed.
Ok, so some patterns are more likely than others, let’s analyse lots and lots of time series data and see if we can build a generalised model that can be fine tuned or used as a feature extractor.
Many time series datasets have repeating patterns, momentum, symmetries, all of these can be learned. Is it perfect? No, but what model is? And things don’t have to be perfect to be useful.
There you go - that’s a pre-trained time series model in a nutshell
I guess they pre-trained the model to exploit common patterns found in any time-series (e.g., seasonalities, trends, etc.)... What would be interesting, though, is to see if it spots patterns that are domain-specific (e.g., the ventricular systole dip in an electrocardiogram), and possibly transfer those (that would be obviously useless in this specific example, but maybe there are interesting domain transfers out there)
My understating is that, while your eye can naturally spot a dependency over time in time series data, machines can’t. So as we did for imaging, where we pre-trained models to let machines easily identify objects in pictures, now we are doing the same to let machines “see” dependencies over time. Then, how these dependencies work, this is another story.
Comments
How can time series model be pre-trained ? I think I’m missing something.
If you have a univariate series, just single values following each other -
[5, 3, 3, 2, 2, 2, 1, …]
What is the next number? Well let’s start with the search space - what is the possible range of the next number? Assuming unsigned 32bit integers (for explanation simplicity) it’s 0-(2^32-1)
So are all of those possible outputs equally likely? The next number could be 1, or it could be 345,654,543 … are those outputs equally likely?
Even though we know nothing about this sequence, most time series don’t make enormous random jumps, so no, they are not equally likely, 1 is the more likely of the two we discussed.
Ok, so some patterns are more likely than others, let’s analyse lots and lots of time series data and see if we can build a generalised model that can be fine tuned or used as a feature extractor.
Many time series datasets have repeating patterns, momentum, symmetries, all of these can be learned. Is it perfect? No, but what model is? And things don’t have to be perfect to be useful.
There you go - that’s a pre-trained time series model in a nutshell
Third paragraph of the introduction of the mentioned paper[1] in the first paragraph of the repo.
[1] https://arxiv.org/abs/2310.10688
I guess they pre-trained the model to exploit common patterns found in any time-series (e.g., seasonalities, trends, etc.)... What would be interesting, though, is to see if it spots patterns that are domain-specific (e.g., the ventricular systole dip in an electrocardiogram), and possibly transfer those (that would be obviously useless in this specific example, but maybe there are interesting domain transfers out there)
My understating is that, while your eye can naturally spot a dependency over time in time series data, machines can’t. So as we did for imaging, where we pre-trained models to let machines easily identify objects in pictures, now we are doing the same to let machines “see” dependencies over time. Then, how these dependencies work, this is another story.