Before anybody thinks that they need something like this at work, I have seen single node HA Prometheus set ups work at one of the largest CDN's in the country for metrics.
Reddit's own Kubernetes infrastructure team uses single node (pod) Promethei as well. [0]
If you look all of the components that are required to run Thanos [1], the operational complexity is incredibly high. I know its a shiny tool, that is super cool but please make sure you have an actual need for some of these before devoting resources to them.
Generally agreed that you can get far with a single Prometheus server (or many independent vanilla Prometheus servers, potentially also using Prometheus's own federation). But I still recommend Thanos as an extension to a lot of people. I like Thanos because it's so easy to deploy alongside an existing Prometheus installation, while itself being mostly stateless (long-term state is kept in object storage), and it gives people:
- a global view over multiple Prometheus servers
- deduplicated view over servers in an HA pair
- durable long-term storage for little cost
The Thanos architecture diagrams (especially the one in their README.md) can look a bit intimidating, but I find it sooo easy to get started with in practice, since you don't even need to deploy all of the components to begin with it. I usually tell people to just drop in a Thanos sidecar next to each of their Prometheus servers, so they will get backups of all their Prometheus server data (for those who are interested in long-term data retention). And then later, they can add the Querier component for an integrated view over multiple servers. And then later they can deploy the Store gateway to also integrate back long-term data into that view. And then at some point, the compactor...
Without being a Thanos expert, it took me ~15 minutes to deploy all those components (+ Minio for object storage) in front of a training audience that wanted to know more about Thanos (while reading Thanos + Minio docs). Of course a proper production deployment always takes way more time, but still I like how conceptually simple it is to integrate Thanos with Prometheus.
Yet Prometheus runs out of steam pretty quickly if one has a need to aggregate metrics from multiple regions or retain metrics for a lengthy period of time.
Comments
Before anybody thinks that they need something like this at work, I have seen single node HA Prometheus set ups work at one of the largest CDN's in the country for metrics.
Reddit's own Kubernetes infrastructure team uses single node (pod) Promethei as well. [0]
If you look all of the components that are required to run Thanos [1], the operational complexity is incredibly high. I know its a shiny tool, that is super cool but please make sure you have an actual need for some of these before devoting resources to them.
[0] https://www.reddit.com/r/kubernetes/comments/ebxrkp/we_are_t...
[1] https://improbable.io/blog/thanos-prometheus-at-scale
Prometheus author here (Julius).
Generally agreed that you can get far with a single Prometheus server (or many independent vanilla Prometheus servers, potentially also using Prometheus's own federation). But I still recommend Thanos as an extension to a lot of people. I like Thanos because it's so easy to deploy alongside an existing Prometheus installation, while itself being mostly stateless (long-term state is kept in object storage), and it gives people:
- a global view over multiple Prometheus servers - deduplicated view over servers in an HA pair - durable long-term storage for little cost
The Thanos architecture diagrams (especially the one in their README.md) can look a bit intimidating, but I find it sooo easy to get started with in practice, since you don't even need to deploy all of the components to begin with it. I usually tell people to just drop in a Thanos sidecar next to each of their Prometheus servers, so they will get backups of all their Prometheus server data (for those who are interested in long-term data retention). And then later, they can add the Querier component for an integrated view over multiple servers. And then later they can deploy the Store gateway to also integrate back long-term data into that view. And then at some point, the compactor...
Without being a Thanos expert, it took me ~15 minutes to deploy all those components (+ Minio for object storage) in front of a training audience that wanted to know more about Thanos (while reading Thanos + Minio docs). Of course a proper production deployment always takes way more time, but still I like how conceptually simple it is to integrate Thanos with Prometheus.
Do you have by any chance recorded this?
No, sorry, it was a private commercial training.
Reddit writes out the metrics to other solution [0], so there is no difference from the setups in the article.
[0] https://www.reddit.com/r/kubernetes/comments/ebxrkp/we_are_t...
Yet Prometheus runs out of steam pretty quickly if one has a need to aggregate metrics from multiple regions or retain metrics for a lengthy period of time.
I think you're conflating the scraping of time-series data with it's storage.
Nope, to be clear I am talking about Prometheus' own TSDB.