This is the last post in the Rust/Tokio/Tracing/OTEL tutorial series.
In the first post, we gave an Introduction to Tokio Tracing and OpenTelemetry.
In the second post, we wrote the code for our application to export structured logs and OTEL spans (to Jaeger) using the Tokio Tracing crate and OpenTelemetry.
In this post, we'll change the code to export traces and logs to GCP instead.
The code for this tutorial can be found in https://github.com/ivan-brko/rust-tracing-otel-tutorial/tree/gcp.
Motivation
If you’re running your application in GCP, we can further improve our solution to make better use of the Observability tools we have available (GCP Log Explorer and GCP Trace Explorer).
We can do the same for Azure and AWS, but that’s not in the scope of this tutorial.
This tutorial assumes that your application is already running in GCP, you can find some resources on how to set it up here: https://www.dotlambda.io/blog/categories/gcp.
What do we want to do differently when running in GCP?
There’s two things we want to change when running on GCP, one for logs and one for traces.
Logs
We’ll still print our logs to the console, but we will use the log format that is supported by GCP Log Explorer. This means printing logs in JSON format + adding a couple of additional metadata fields that GCP can interpret.
The full list of supported metadata fields can be found here https://cloud.google.com/logging/docs/structured-logging. For example, we’ll decorate our logs with OTEL trace id under key logging.googleapis.com/trace which will allow GCP to correlate the log with the traces we export to GCP Trace Explorer.
Traces
Instead of exporting the traces to Jaeger, we’ll export the traces to GCP Trace Explorer.
Needed code changes
Let’s start with Cargo.toml:
We removed the Jaeger exporting dependency and added two new dependencies. tracing-stackdriver is used to print logs in the format GCP expects them, and opentelemetry-stackdriver is used to export traces to GCP.
Next we want to add a new file to our src/ called custom_span_processor.rs. This file is not necessary to export the traces or logs correctly, but it will add the attribute to our span that will be displayed as our service name in the GCP traces. It’s a simple file, we implement the trait SpanProcessor, and only make use of the on_start() method to add the service name to our span whenever a new span is created. Note that we are adding an attribute on our OTEL context, not our Tracing context.
Most of the changes are in our init.rs file where we change the log format we used and our OTEL exporter:
Finally, just some minor changes are needed in our main.rs:
If we now deploy this latest version of our application to GCP, we should be able to see our traces when we open the Trace Explorer in GCP Console:
Notice that spans last exactly as long as we expected them to, given the sleeps we configured and that spans exported by our application have the correct service name, set to the value we configured in our CustomSpanProcessor.
The small white circles in the spans indicate log messages correlated with that span, so you can easily inspect all the logs belonging to a span.
Also, in our Log Explorer we can filter by trace id and see all the logs belonging to a single request/trace:
Conclusion
That is the end of this tutorial series.
I hope reading it was as helpful to you as writing it was for me.
Comments