Ask HN: Distributed tracing service with critical path analysis?

10 pointspseudonom-9 years ago5 comments

The Dapper paper mentions "An engineer working on long tail latency debugging built a small library which infers hierarchical critical paths from DAPI Trace objects. These critical path structures were then used to diagnose problems and prioritize prospective performance improvements for universal search."

Do any of the distributed tracing services (e.g. Datadog APM, AWS X-Ray) offer this? I don't see any services which tout this functionality.

itamarst 9y

See if you can find a tool that creates flamegraph input files from the traces, or if not write one yourself. My causal logging library, Eliot (https://eliot.readthedocs.io), has an experimental branch for that, and it wasn't particularly hard to create the input files.

https://github.com/brendangregg/FlameGraph will then create nice graphs that can help spot bottlenecks.

albertwang 9y

Disclosure: I work at Datadog --

Tracing requests across services and hosts to identify bottlenecks is one reason why Datadog decided to build a tracing component in APM! A flamegraph is one such method to visually infer critical path analysis that Datadog APM supports. Some obvious things to consider are ease of setup and whether the libraries and languages you use are supported. Some not-so-obvious things to consider are:

  (1) whether you wish to infer this from a single request, or across multiple requests, 
  (2) how to define "interesting" traces -- as you could be collecting millions or billions of traces in production 
  (3) how to create a bigger, "aggregate" picture across individual traces.

We're working on additional ways to easily identify critical paths as well.

programd 9y

I think what you're looking for is Zipkin [0]. We use it in our microservers at Nulladmin.com via the opentracing Go library [1] and the zipkin-go-opentracing library [2]

[0] https://github.com/openzipkin/zipkin

[1] https://github.com/opentracing/opentracing-go

[2] https://github.com/openzipkin/zipkin-go-opentracing

pseudonom-OP9y

My company actually has and uses Zipkin. The thing that I'm looking for that I think is lacking is a way to identify critical paths without manually clicking through a bunch of traces and eyeballing them.

jjirsa 9y

+1 for zipkin

As a Cassandra fan, bonus points that it integrates all the way down into the Cassandra internal request tracing features, afaik the only tracing app/library that does so.

Comments