This article is a write up of a talk that I will give at the osmc in Germany in November about Distributed tracing. It is a sequence of questions I got about distributed systems and distributed monitoring.
It always depends, I find distributed tracing useful in a microservices environment or more in general when there is a request that flies in a system crossing different applications, queues or processes. If you have a problem understanding where a request fails, you need to follow it in some way and tracing does just that.
First of all, we should probably change the name request, it looks too HTTP oriented, and it is not really what we look for now. In modern application, you are interested in events. You need to monitor an event:
These are all events, and probably in your system, they are distributed not via
HTTP but maybe they go in a queue, or they are broadcasted using Kafka or Redis.
Distributed tracing is all about tracking events. The way to go is to create an
id. Usually, it is called
trace_id and you need a way to
propagate it in your system.
For example, in a queue, you can put the
trace_id as part of the payload. Via
HTTP or gRPC you can use Headers.
Your application can take that id, and it can create the span to trace a particular section.
In my mind, this is a picture for one trace. Every segment is a span.
So, every span has a trace id, and every span has its own
You can attach information to every span as key value store. Let’s suppose a
span represent a query in mysql you can put the query as metadata in the span
itslef. In this way you will have a bit more context.
I can’t convince you that interoperability is essential if you already analyzed the problem and you answered “No” to yourself. To build a trace you need to agree on something over languages and applications. That’s why I think a standard is something you can not avoid, at the end you will end up having one just for your company.
The applications that are writing traces is not important. Traces is cross platform and languages. Usually, you point an app to a tracer. It can be Zipkin, Jaeger or others.
The tracer takes all the traces, and it stores them in a storage. The databases are usually ElasticSearch, Cassandra, InfluxDB. It depends on which tracer you are using. They support different databases.
In general traces are high cardinality oriented data, and you can write a lot of them in a short amount of time. So it is a write-intensive application.
There are a couple of other pieces that you can add in your tracing infrastructure:
trace_idis enough for example.
I am an interoperability oriented developer; I think it is essential to avoid vendor lock-in and embracing a big community like the opentracing one you get a lot of tools and services already instrumented with this protocol. It makes my life easy.
You can; there are a couple of tracers open source.
Both of them are open source, and they support different backends like ElasticSaerch, Cassandra and so on.
There are, NewRelic has an opentracing compatible API, or Lightstep for example. A lot of cloud providers offer a tracing service. AWS X-RAY or Google Stackdriver.
You can, but they are a high cardinality data. The
trace_id is usually the
lookup parameter for your queries. It means that it should be indexed, but it
changes for every request. The consequence is a big index.
You need to keep it in mind.
I left this question as the last one because I read it in the opentracing mailing list and I think it is a hilarious question.
First of all, you don’t buy a pan and after the fact you start asking yourself why you have it.
Probably you need to write something, and for that reason, you buy a pen.
Anyway, I trace my applications because it helps me to understand my environment over the “distribution complexity.” I can detect what is taking too long and a trace helps me to understand what to optimize.
Opentracing has a set of standard annotation very useful to detect network latency between services. You can mark a span as “client send” request for example. And when the server gets the request, you can mark another span as “server received.” This two information is useful to know how much time your request spends going from client to the server and you can optimize them time usually working on the proximity between these two applications.
More in general you can parse a trace to get what ever you need as normal logs or events the powerful things is downsampling and analysis. If you are tracing a queue system you can get the average time for a worker to process a message.
Let me know if you have more questions on twitter @gianarb. I am happy to answer them here.