the zero injection fix for sparse counters is the most underrated part of this writeup -- injecting a synthetic zero on first flush to anchor the cumulative baseline is actaully a pretty elegant solution to a problem that bites almost every team migrating from delta-based systems to prometheus, and the fact that they centralized it in the aggregation tier rather than pushing the fix to every instrumentation callsite is exactly the right call.
There is another approach for solving this issue - to use increase_pure() function from MetricsQL - https://docs.victoriametrics.com/metricsql/#increase_pure . Of course, you need to switch to VictoriaMetrics, since Mimir doesn't support this function.
Comments
the zero injection fix for sparse counters is the most underrated part of this writeup -- injecting a synthetic zero on first flush to anchor the cumulative baseline is actaully a pretty elegant solution to a problem that bites almost every team migrating from delta-based systems to prometheus, and the fact that they centralized it in the aggregation tier rather than pushing the fix to every instrumentation callsite is exactly the right call.
There is another approach for solving this issue - to use increase_pure() function from MetricsQL - https://docs.victoriametrics.com/metricsql/#increase_pure . Of course, you need to switch to VictoriaMetrics, since Mimir doesn't support this function.
I was under impression that problem of zero injection was solved with Start Timestamp from OpenMetrics 2.0 spec - see https://prometheus.io/docs/specs/om/open_metrics_spec_2_0/#s...