Microservices: Zipkin, distributed tracing

This is the 5th part of the Developing Microservices with Spring Boot series. After being able to create an edge service, compose services on Zuul and monitor the proxy connections with Hystrix dashboard, we will have a look to how to create traces that can be monitored.
This will let us to visualize a call from beginning to end, analyze timeouts, etc…
Being able to trace any request is a nice to have feature in our microservice infrastructure.  The full code is available on github.

Overview

Based on the previous examples, the architecture we will work with is the following:

Microservices- Zipkin server

Microservices- Zipkin server

For this post, the incorporation of the Zipkin stream server will allow us to receive http traces that later can be shown through the Zipkin Web UI.
After reading the 8 fallacies of distributed computing,  you definitely want some monitoring or analysis tool that tells you what is going on with your http requests.
The more complex the system is, the more need of tools for analyzing what is going on on the communication between services.

For the purpose of the post, we will analyze with Zipkin a composition of services happening on the Zuul service:

Microservices- Composition and http tracing

Microservices- Composition and http tracing

The Zuul service will expose an endpoint /profiles/names.
When performing the request, the micro-proxy will get data from Service A and Service B. After that, it will merge it and return it to the client.
On the previous picture the sequence of the requests is described. Zipkin will be able to represent the trace of the request in a similar way to the following table:

Date Time Relative Time Service Annotation Host
3/5/2016, 8:48:08 PM zuul-service Client Send 192.168.1.138:9000
3/5/2016, 8:48:08 PM 2.000ms svca-service Server Receive 192.168.1.138:5000
3/5/2016, 8:48:08 PM 10.000ms svca-service Server Send 192.168.1.138:5000
3/5/2016, 8:48:08 PM 11.000ms zuul-service Client Receive 192.168.1.138:9000
3/5/2016, 8:48:08 PM 11.000ms zuul-service Client Send 192.168.1.138:9000
3/5/2016, 8:48:08 PM 13.000ms svcb-service Server Receive 192.168.1.138:7000
3/5/2016, 8:48:08 PM 22.000ms svcb-service Server Send 192.168.1.138:7000
3/5/2016, 8:48:08 PM 22.000ms zuul-service Client Receive 192.168.1.138:9000

Hands on code

For this post we will focus on creating a Zipkin server and the changes required on the microservices in order to start sending traces to the server.

We will have our Zipkin server initialized through the configuration microservice. Bootstrapping the service will allow us to control all the initial configuration from one place.

Configuration service

File: configuration-service/src/main/resources/config/zipkin-server.properties

Apart from adding the properties for the new Zipkin server, the properties of the current microservices will need some additions related to logging. The idea is to be able to identify any input/output through http on the http headers, but also on the logs.

The logging of the applications will be something like:

The change required for the services A and B configuration is the addition of logging.pattern.console:

File: configuration-service/src/main/resources/config/svca-service.properties

At the same time, we would need to do the same for the Service B:

File: configuration-service/src/main/resources/config/svcb-service.properties

And the Zuul proxy:

File: configuration-service/src/main/resources/config/zuul-service.properties

Zipkin server

In order to deploy a Zipkin server, there are several ways to do it. In our case, we will create a standalone service that will listen to a ‘queue‘ of events (in our case in Redis) and will update the in-memory information. This is just to avoid a complex setup.

For production, the recommended way is having a data storage solution hooked up to the Zipkin Stream Server (for more information you can have a look to the OpenZipkin project).

In order to create the standalone Zipkin Stream server we will need to add the following dependencies to our pom.xml

As the Zipkin stream server is bootstrapped, instead of application.properties, we will have the bootstrap one (zipkin-server/src/main/resources/bootstrap.properties)

On the main configuration file we will need to add the @EnableZipkinStreamServer annotation:

After setting up the main class, we should be able to deploy the zipkin-server project with mvn spring-boot:run

Changes on current microservices

The services that will inject information on the http headers relate to the request are:

  • Zuul service: the proxy
  • Service A: returns the profiles associated to this service
  • Service B: returns the profiles associated to this service

The changes required on these services are related to:

  • Dependencies: pulling out the artifact required to write into the http headers
  • Configuration: we need to specify which http requests get marked. For our case, we will sample each of the http requests.

Dependencies

Apart from the configuration details we did before, we would need to add the dependency to all the required pom.xml:

Configuration

Each of the services will need to specify what they want to sample. Zipkin starter provides you a Sampler class that puts you in control of how may requests you want to inject:

These files are the same for the service A and B. All the configurations can be found on the project on the following files:

After setting up the configuration files and created the Zipkin Stream server we should be able to deploy everything.

Deployment

After having done the modifications to the projects (or checked out the code), we can run each of the services with the following:

mvn spring-boot:run

For this example, to run everything, the recommended way to deploy is:

  • Deploy Redis. We use it for the cloud streaming messages propagated to the zipkin stream server.
  • Deploy config-service
  • Deploy service a
  • Deploy service b
  • Deploy Eureka service
  • Deploy Zuul proxy service
  • Deploy Zipkin server
  • Deploy Zipkin web ui pointing to the Zipkin stream server:
    • Download the Zipkin web ui from the build . For this example, I downloaded zipkin-web-1.28.1-20160104.090947-7-all.jar , although you could use any newer version.
    • Note this step could be replaced if you install the Zipkin with docker. You could have storage, ui and the server running for production in no time. For more information, have a look to OpenZipkin.

Running jps should give us something like:

We could also check some information that the cloud stream project generates on redis. Connecting to redis through redis-cli we can see it creates some keys:

 

Testing and interacting with Zipkin

Once deployed everything, we can start generating some traces. In order to do that, hit the following url a couple of times:

After a couple of seconds, the traces should be already on Zipkin. In order to check that, we should be able to access to the web ui on http://localhost:9401/ :

Http traces performed on Zipkin Web ui

Http traces performed on Zipkin Web ui

Clicking on one of the traces, it whould give us detailed information of the different services hit:

Http trace expanded with Zipkin Web UI

Http trace expanded with Zipkin Web UI

This is a good way to see possible timeouts, bottlenecks, etc…

If interested on the complete trace of the request, clicking on the main trace should give us the timings and hops of the request inside our microservice architecture:

Http trace detailed with Zipkin

Http trace detailed with Zipkin

Summary

  • In this post it was showcased how we could use Zipkin in our microservice infrastructure. On distributed systems, http tracing is important as main communication between processes happens through the network (different machines)
  • Http tracing becomes really useful for analyzing time responses, timeouts, bottlenecks, http overheads… Zipkin Web UI provides a simple and clean way to get into details of
  • For having Zipkin deployed into your production environment, better make sure to have some storage for future lookups. OpenZipkin provides a variety of docker images you can use for different purposes.

Leave a Reply

Your email address will not be published. Required fields are marked *