When a request goes gateway → booking-service → payment-service → Kafka → notification-service, every service writes its own log, the clocks disagree by a few milliseconds, and nothing tells you which log line belongs to which request. Distributed tracing gives every request an id that travels with it through every service, so you can answer “why did this request take 3 seconds?”. By the end you’ll be able to read a traceparent header, attach the Java agent, choose between head and tail sampling, and link logs to traces.
This post starts from the tracing diagram of the course’s homestay project (Spring Boot + the OpenTelemetry Java agent + SigNoz), extended from the official OpenTelemetry and W3C docs. Related: Reverse proxying with Nginx (passing a request id through the proxy), Error handling in Spring Boot (returning a traceId in error responses).
1. Three signals and where traces fit
Key idea: metrics tell you something’s wrong, a trace shows which request was slow and where, and the logs of that exact span give the details. The trace id ties all three together.
OpenTelemetry (OTel) covers three kinds of signals (telemetry: data a system reports about itself):
| Signal | Answers | Example |
|---|---|---|
| Metrics | Is the system healthy, what’s the trend? | p99 latency, errors per second |
| Logs | What happened at this particular point? | “payment declined, code=51” |
| Traces | Where did this request go, where was it slow? | the span tree of one request |
2. Traces and spans
Key idea: a trace is a request’s whole journey; a span is one piece of work in it that knows its parent. Spans sharing a trace id form a tree, drawn as a waterfall chart.
A trace is identified by a 16-byte trace_id. A span has:
- a
span_id(8 bytes) and aparent_span_id(the root span has no parent); - a name, a start time and an end time;
- attributes: key-value pairs describing the work (
http.request.method,http.route,db.system); - events: timestamped points inside the span (an exception, for example);
- links: references to causally related spans in other traces (common with batch consumers, where one span processes messages from many traces);
- a status:
Unset,OkorError; - a span kind:
Server(handles a synchronous request),Client(makes a synchronous call),Producer(hands off work to be done later),Consumer(processes that work),Internal(in-process work, the default).
The course diagram, redrawn: the agent in Service 1 creates the root span and sends the context to Service 2 in a header, Service 2 creates a child span; both export spans to the Collector, which writes them to storage for the UI to query.
sequenceDiagram
participant C as Client
participant S1 as Service 1 (agent)
participant S2 as Service 2 (agent)
participant OC as OTel Collector
participant DB as ClickHouse
participant UI as SigNoz UI
C->>S1: 0. HTTP request
Note over S1: 1. create span: trace t1, span s1, parent null
S1->>S2: 2. HTTP + traceparent (t1, s1)
Note over S2: 3. create child span: trace t1, span s2, parent s1
S2-->>S1: 4. response
S1-->>C: 5. response
S1-)OC: 6. export spans over OTLP
S2-)OC: 6. export spans over OTLP
OC->>DB: write spans
UI->>DB: query trace t1
In the UI a trace shows up as a waterfall:
trace 0af7651916cd43dd8448eb211c80319c 0ms ......... 820ms
└─ GET /bookings (gateway) SERVER [=============================] 820ms
└─ POST /bookings (booking-service) SERVER [===========================] 790ms
├─ SELECT homestay_availability ... CLIENT [==] 45ms
├─ POST /payments (payment-service) CLIENT [====================] 610ms
│ └─ POST /payments SERVER [===================] 600ms
│ └─ POST vnpay.example/init CLIENT [=================] 570ms
└─ send booking-created PRODUCER [=] 8ms
One look tells you: 610 of the 790 ms are in payment, nearly all of it waiting on the external payment gateway. When the latency sits with a third party like this, what to log for every outbound call and inbound webhook, how to pass traceparent to a partner, and when to alert as p99 approaches their timeout are covered in Third-party integration.
3. Context propagation and the traceparent header
Key idea: the caller sends the context in the W3C traceparent header, four fields version-traceid-parentid-flags. A malformed header is silently ignored and the trace splits in two.
Context propagation means carrying the current trace id and span id across a boundary (a service, a thread, a message). The common standard is W3C Trace Context: the traceparent and tracestate headers. The slide writes traceparent: <trace_id>-<span_id>; that’s a shorthand, and the real format is:
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
│ │ │ └ trace-flags (01 = sampled)
│ │ └ parent-id: 16 hex (8 bytes), the caller's span id
│ └ trace-id: 32 hex (16 bytes)
└ version: 00
- An all-zero
trace-idorparent-idis invalid; the receiver ignores the header and starts a new trace. - The lowest bit of
trace-flagsis the sampled flag: the caller may have recorded this trace. The receiver uses it to decide whether to record too (theparentbasedsamplers). tracestatecarries vendor-specific data as comma-separatedkey=valuepairs, up to 32 of them.- OTel also has a
baggageheader for application key-value pairs (saytenant=acme) that travel through every service.
A hand-rolled traceparent parser and generator in plain Java 21
For learning only; real code uses OTel’s W3CTraceContextPropagator. I ran it on JDK 21:
import java.security.SecureRandom;
import java.util.HexFormat;
import java.util.Optional;
import java.util.regex.Pattern;
public record TraceParent(String traceId, String spanId, boolean sampled) {
private static final Pattern FORMAT =
Pattern.compile("^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$");
private static final SecureRandom RANDOM = new SecureRandom();
private static final HexFormat HEX = HexFormat.of();
/** Parse an incoming header; empty means "start a new trace". */
public static Optional<TraceParent> parse(String header) {
if (header == null) return Optional.empty();
var m = FORMAT.matcher(header.trim());
if (!m.matches() || m.group(1).equals("ff")) return Optional.empty();
String traceId = m.group(2), spanId = m.group(3);
if (traceId.chars().allMatch(c -> c == '0') || spanId.chars().allMatch(c -> c == '0')) {
return Optional.empty();
}
boolean sampled = (Integer.parseInt(m.group(4), 16) & 0x01) == 1;
return Optional.of(new TraceParent(traceId, spanId, sampled));
}
public static TraceParent newRoot(boolean sampled) {
return new TraceParent(randomHex(16), randomHex(8), sampled);
}
/** Child span: same trace id, new span id; the caller's span id becomes our parent. */
public TraceParent child() {
return new TraceParent(traceId, randomHex(8), sampled);
}
public String header() {
return "00-" + traceId + "-" + spanId + "-" + (sampled ? "01" : "00");
}
private static String randomHex(int bytes) {
byte[] b = new byte[bytes];
do { RANDOM.nextBytes(b); } while (allZero(b));
return HEX.formatHex(b);
}
private static boolean allZero(byte[] b) {
for (byte x : b) if (x != 0) return false;
return true;
}
public static void main(String[] args) {
String incoming = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01";
var parent = parse(incoming).orElseGet(() -> newRoot(true));
var child = parent.child();
System.out.println("parent " + parent.header());
System.out.println("child " + child.header() + " (parent span " + parent.spanId() + ")");
System.out.println("bad1 " + parse("00-00000000000000000000000000000000-b7ad6b7169203331-01"));
System.out.println("bad2 " + parse("00-0AF7651916CD43DD8448EB211C80319C-b7ad6b7169203331-01"));
System.out.println("simple " + parse("0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331"));
}
}
Output:
parent 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
child 00-0af7651916cd43dd8448eb211c80319c-6b6d3322f8c71382-01 (parent span b7ad6b7169203331)
bad1 Optional.empty
bad2 Optional.empty
simple Optional.empty
It’s stricter than the spec in one way: the spec asks receivers to try parsing headers with a version above 00 using the first four fields, while this code accepts only the version-00 format.
Where context gets lost
Inside a process, the context lives in a ThreadLocal (OTel’s Context.current()). It’s lost wherever work jumps to another thread or leaves the process without someone carrying it:
ExecutorService,CompletableFuture.supplyAsync,@Async: the Java agent wraps common executors for you; without it, useContext.current().wrap(runnable)orContext.taskWrapping(executor).- Message brokers: the context has to go into the message headers. The agent does this for the Kafka client, RabbitMQ and JMS. With Kafka, the consumer span’s parent is the producer span, so one trace covers the asynchronous part too.
- Scheduled jobs and batches reading an outbox table: there’s no incoming request, so each run is a new trace. To connect it with the original request, store the
traceparentin the outbox row and use a span link when processing it.
4. OpenTelemetry architecture
Key idea: your code depends only on the OTel API; the SDK or the Java agent creates spans and sends them over OTLP to a Collector, which batches, filters and samples before forwarding to storage.
flowchart LR
subgraph App["Java application"]
API["OTel API<br/>(called by code)"] --> SDK["OTel SDK<br/>sampler, processor, exporter"]
AG["Java agent<br/>auto-instruments Spring, JDBC, Kafka..."] --> SDK
end
SDK -- "OTLP gRPC :4317<br/>or HTTP :4318" --> COL["OTel Collector<br/>receivers → processors → exporters"]
COL --> ST[("Storage<br/>ClickHouse / Tempo / Jaeger")]
ST --> UI["UI<br/>SigNoz / Grafana"]
- API: the interfaces for creating spans and reading context. Libraries depend on the API only, never on the SDK.
- SDK: the implementation: sampler, span processor (usually
BatchSpanProcessor), exporter. - Instrumentation: code that creates spans for existing libraries. The Java agent (zero-code) injects bytecode at class-load time for hundreds of libraries: Spring MVC/WebFlux, JDBC, Hibernate, Kafka, RestTemplate/WebClient, gRPC, Redis clients.
- OTLP: the wire protocol, over gRPC (port 4317) or HTTP/protobuf (port 4318).
- Collector: a separate process that receives, processes and forwards telemetry. A pipeline is receivers (OTLP, Jaeger, Prometheus), processors (memory limiter, batch, sampling, removing sensitive attributes) and exporters (to one or more backends).
- Backend: SigNoz (stores in ClickHouse, a column store that suits wide, write-heavy span data read by time range), Jaeger, Grafana Tempo, or a paid service.
Attaching the Java agent
The course’s approach: download opentelemetry-javaagent.jar and configure it through environment variables so the start command doesn’t change:
export JAVA_TOOL_OPTIONS="-javaagent:/otel/opentelemetry-javaagent.jar"
export OTEL_SERVICE_NAME=booking-service
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=1.4.2
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1
java -jar booking-service.jar
Remember: protocol and port must match, and the default sampler records 100% of traces, far too expensive for a busy production system (section 5).
Java agent defaults according to the OTel docs
- Since 2.0, the Java agent exports OTLP over
http/protobufby default (port 4318); before that it wasgrpc(4317). The course README setsgrpcwith endpoint:4317: still correct, as long as protocol and port match. - Traces, metrics and logs all default to the
otlpexporter; the default propagators aretracecontext,baggage. - The default sampler is
parentbased_always_on: it records 100% of traces, fine in development and far too expensive for a busy production system. - A Collector isn’t required: the SDK can send straight to a backend. The OTel docs still recommend running a Collector next to your services so they can offload data quickly and leave retries, batching and filtering of sensitive data to it.
Your own spans
The agent already creates spans for HTTP and the DB; add spans yourself for important business steps, with @WithSpan or the OTel API. Call span.end() in a finally: forget it and the span is never exported.
Creating spans with @WithSpan and with the OTel API
It needs opentelemetry-api and opentelemetry-instrumentation-annotations. I didn’t run this one because it needs a whole application plus the agent.
@Service
@RequiredArgsConstructor
class PricingService {
private final Tracer tracer; // GlobalOpenTelemetry.getTracer("booking") or a bean
// Option 1: annotation; the agent creates a span named "PricingService.quote"
@WithSpan
Money quote(@SpanAttribute("homestay.id") long homestayId, DateRange nights) {
return compute(homestayId, nights);
}
// Option 2: the API, with control over name, attributes, events and status
Money quoteManual(long homestayId, DateRange nights) {
Span span = tracer.spanBuilder("pricing.quote")
.setAttribute("homestay.id", homestayId)
.setAttribute("pricing.nights", nights.count())
.startSpan();
try (Scope ignored = span.makeCurrent()) { // child of the current request span
Money total = compute(homestayId, nights);
span.setAttribute("pricing.total_vnd", total.amount().longValue());
return total;
} catch (RuntimeException e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR, e.getMessage());
throw e;
} finally {
span.end(); // forget end() and the span is never exported
}
}
}
5. Sampling: how much to keep
Key idea: keeping everything costs more than the system being observed, so you sample (keep only part of the traces). Head sampling decides at the start of the trace, cheap but blind to errors; tail sampling decides in the Collector after seeing the whole trace.
A quick estimate: 2,000 requests/s × 20 spans × 1 KB per span (compressed) = 40,000 spans/s, 40 MB/s, about 3.5 TB a day. And most traces look alike (fast, successful).
| Head sampling | Tail sampling | |
|---|---|---|
| Where and when it decides | the first service, when the trace starts, from the trace id | the Collector, after seeing (most of) the trace |
| Example | traceidratio at 0.1 keeps 10% |
keep every error, every trace > 2 s, every payment trace, 5% of the rest |
| Pros | cheap, simple, stateless; the sampled flag travels in traceparent, so services using parentbased_* follow the root’s decision and traces have no holes |
keeps exactly the traces worth looking at |
| Cons | decides before knowing whether the request fails or is slow: 90% of failing traces are dropped too | the Collector holds spans in memory while waiting (decision_wait), it’s stateful, and all spans of a trace must reach the same Collector instance |
A tail-sampling Collector config that was actually run
I checked the config below with otelcol-contrib validate (version 0.114.0) and ran it: I sent four traces (one error, one slow at 3 seconds, one from payment-service, one fast and successful), and the Collector exported only the first three.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter: # first in the pipeline: refuse data rather than get OOM-killed
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 20
tail_sampling:
decision_wait: 10s # wait for the trace's spans to arrive
num_traces: 50000 # traces held in memory
policies: # keep the trace if ANY policy matches
- name: keep-errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: keep-slow
type: latency
latency: { threshold_ms: 2000 }
- name: keep-payments
type: string_attribute
string_attribute: { key: service.name, values: [payment-service] }
- name: sample-the-rest
type: probabilistic
probabilistic: { sampling_percentage: 5 }
batch:
send_batch_size: 8192
timeout: 2s
exporters:
otlp/backend:
endpoint: signoz-otel-collector:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, tail_sampling, batch]
exporters: [otlp/backend]
6. Linking logs to traces
Key idea: the Java agent puts trace_id, span_id and trace_flags into the Logback/Log4j MDC; print them and you can go from a span to exactly its log lines.
The MDC (Mapped Diagnostic Context) is a per-thread bag of key-value pairs that the logger adds to each line.
<!-- logback-spring.xml -->
<pattern>%d{ISO8601} %-5level [%X{trace_id},%X{span_id}] %logger{36} - %msg%n</pattern>
Better still, log JSON (Spring Boot 3.4 has built-in structured logging: logging.structured.format.console=ecs or logstash) and ship it over OTLP or a log agent, so a span in the UI links straight to its logs. Two more places where the trace id earns its keep:
- Error responses: return the
traceIdin the error body (ProblemDetail) so users or the support team can send it back and engineers can open the trace directly. See Error handling in Spring Boot. - Exemplars: histogram metrics (latency, say) carry the trace ids of a few samples; click the spike on the p99 chart and you land on a real trace.
7. Running it in production
Key idea: four things to handle in production: clock skew, sensitive data, CPU/heap cost in the service, and retention.
- Clock skew: spans from two machines use two clocks, and a few milliseconds of skew makes a child span look like it started before its parent. Run NTP/chrony; when reading a waterfall, trust each span’s duration more than the relative position of spans from different machines.
- Sensitive data: instrumentation can record SQL (parameters are sanitised by default), URLs with tokens in the query string, headers. Filter in the Collector with the
attributes/redactionprocessors before data leaves your infrastructure. - Cost in the service: the agent adds a few percent of CPU and some tens of MB of heap; measure before and after. Disable instrumentations you don’t need (
OTEL_INSTRUMENTATION_<NAME>_ENABLED=false) if a library produces lots of tiny spans. How to measure before and after properly (same load, compare p99, not averages) is in Performance testing. - Retention: traces are valuable for days to a few weeks; set a TTL on the ClickHouse tables instead of keeping them forever.
Exercises
Exercise 1: Read and produce a traceparent
Task: the gateway receives traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 and calls booking-service. (a) What are the trace id and the caller’s span id, and is it sampled? (b) What header does the gateway send to booking-service? (c) What if the incoming header is 00-4bf92f3577b34da6a3ce929d0e0e4736-0000000000000000-01?
Solution to Exercise 1
(a) Trace id 4bf92f3577b34da6a3ce929d0e0e4736, parent span id 00f067aa0ba902b7, flags 01, so it’s sampled.
(b) The gateway creates its own span with a new span id, say 53995c3f42cd8ad8, and sends 00-4bf92f3577b34da6a3ce929d0e0e4736-53995c3f42cd8ad8-01: same trace id, and the parent-id is the gateway’s span id, not 00f067aa0ba902b7.
(c) An all-zero parent-id is invalid: the gateway ignores the header and starts a new trace with a new trace id. If you see this often, some client or proxy upstream is generating broken headers.
Exercise 2: A sampling policy for a booking system
Task: the system peaks at 3,000 requests per second, the budget is about 300 GB of traces per day, traces average 15 spans at 1 KB each. Requirements: never lose a failing trace, be able to see slow requests, keep payment flows complete. Design the sampling.
Solution to Exercise 2
Keeping everything takes 3,000 × 15 × 1 KB = 45 MB/s, about 3.9 TB/day at peak, more than ten times the budget.
Use tail sampling in the Collector as in section 5: keep 100% of traces with errors, 100% over 2 seconds, 100% touching payment-service, and 5% of the rest. If errors and slow requests make up about 2% and payments about 3%, you keep roughly 5% + 0.95 × 5% ≈ 10%, about 380 GB/day if the whole day ran at peak. The daily average is lower, so it fits.
Deployment:
- Tier 1: a Collector agent (sidecar or DaemonSet) with a
loadbalancingexporter keyed ontraceID. Tier 2: a pool of Collectors runningtail_sampling. - The SDK stays at
parentbased_always_onso every span reaches the Collector. - Watch the
otelcol_processor_tail_sampling_*metrics and Collector memory; if it gets overloaded, add 50% head sampling at the gateway.
Exercise 3: The trace breaks at Kafka
Task: in SigNoz, the POST /bookings trace stops at the send booking-created span; notification-service’s processing of the message shows up as a separate trace. List the possible causes and how to check each.
Solution to Exercise 3
- notification-service doesn’t run the agent, or Kafka instrumentation is disabled: check the agent’s startup log.
- The message has no
traceparentheader: look withkafka-console-consumer --property print.headers=true. A common cause: the producer sits behind custom code (an outbox relay reading from the DB and sending, for example), so the original request’s context is gone. Fix it by storingtraceparentin the outbox row and setting it on the header when sending. - The consumer processes batches: one span handles messages from many traces, so the instrumentation uses span links instead of a parent-child relation. In the UI you follow the link to reach the original trace.
Remember
- Can you write out the four fields of
traceparent, and explain why the<trace>-<span>shorthand breaks traces? - Can you name the three places context usually gets lost (thread pools, brokers, outbox/jobs) and how to carry it?
- Can you compare head and tail sampling: where each decides, what it costs, why you route by trace id?
- Can you explain why span names must be route templates and why baggage must never hold personal data?
- Do you know the path from a bug report to the cause:
traceIdin the response → trace → logs via the MDC?
Interview questions
Junior
What’s the difference between a trace and a span?
A trace is a request’s whole journey (one trace id); a span is one piece of work inside it.
A span (an HTTP call, a query, a message send) has a start, an end, attributes and a parent span. Spans sharing a trace id form a tree that shows where the time went.
What are logs, metrics and traces each for?
Metrics say whether the system is healthy, traces show where one request was slow, logs give the details at one point.
Metrics (p99, error rate) are cheap to keep for a long time. You typically go from a metric alert to a trace to narrow things down, then to the logs of that span; the trace id ties the three together.
Mid
How is context passed between services?
Through W3C Trace Context headers: traceparent in the form version-traceid-parentid-flags, plus tracestate for vendor data.
For example 00-<32 hex>-<16 hex>-01. The caller puts its own span id in parent-id, and the receiver creates a child span with the same trace id. With message brokers, the context goes in the message headers. Within a process it lives in a ThreadLocal, so it has to be carried along when work moves to another thread.
What does the OpenTelemetry Java agent do?
It injects bytecode at class-load time to create spans for common libraries automatically, with no code changes.
It covers Spring MVC, JDBC, Kafka and HTTP clients; propagates context through headers and executors, puts the trace id into the logging MDC, and exports traces, metrics and logs over OTLP. It’s configured with OTEL_* environment variables. The price is slower startup and no native-image support.
Why run an OTel Collector instead of exporting straight to the backend?
So services only have to push data quickly, while batching, retries, filtering, sampling and routing happen in the Collector.
The Collector also enforces memory limits, scrubs sensitive data, runs tail sampling and sends to one or several backends. Switching backends doesn’t require redeploying services. For small systems or development, exporting directly is fine.
A child span starts before its parent in the waterfall. Why?
The two machines’ clocks are skewed.
Each span’s duration is still right because it’s measured on one machine, but their relative positions are off by the skew. Run NTP/chrony everywhere and don’t draw conclusions from a few milliseconds of offset between services.
Senior
How do head and tail sampling differ, and which do you pick?
Head decides at the start of the trace from the trace id, cheap but blind to errors; tail decides in the Collector after seeing the trace, keeps errors but is costly and stateful.
- Head: the decision travels with the sampled flag so traces stay complete, but it can’t know whether the request will fail or be slow.
- Tail: every span must reach the Collector, the Collector needs memory, and you must route by trace id so all spans of a trace reach the same instance.
- Low volume: keep 100% with short retention. High volume: tail sampling keeps errors, slow and critical flows and a small share of the rest, with head sampling in front for protection if needed.
How do you keep a trace intact across Kafka and an outbox?
Carry traceparent in the message headers; with an outbox, store it in the outbox row so the relay can set it back.
The producer puts traceparent in the headers (the agent does this for the Kafka client), and the consumer reads it and creates a child span or a span link. With an outbox, the original request only writes to the DB and a relay sends the message later on another thread without the context, so the relay must take traceparent from the row. Batch consumers use span links because one span can’t have several parents.
Where does tracing cost you performance or money, and how do you control it?
CPU, heap and startup time in the service, plus network and storage per span; control it with measurement, sampling, low cardinality and TTLs.
- In the service: CPU and heap for instrumentation, agent startup time, the export queue. In the network and storage: spans times requests.
- Controls: measure before and after enabling, disable instrumentations that produce tiny spans, sample, use templated span names, set TTLs in storage, and monitor dropped spans.
- Export is asynchronous and batched, so a dead Collector doesn’t slow requests down; the trade-off is that spans get lost.
A user reports a failed request. How do you get from the report to the cause?
Take the traceId from the error response, open the trace to find the Error span, then read that span’s logs via the trace_id in the MDC.
The error response includes a traceId (ProblemDetail), and the user or support sends it to us. In the trace, I look for the span with Error status and the spans taking the time. That’s why the tail-sampling policy must always keep failing traces. Even without a trace, the trace id still filters logs across every service, because logs carry it whether or not the span was sampled.
Useful phrases for interviews
- “Metrics told us something was wrong; the trace told us where; the logs of that span told us why.”
- “Context propagation breaks whenever work hops to another thread or goes through a queue without headers.”
- “We use parent-based sampling so every service respects the decision made at the entry point.”
- “Tail sampling lets us keep every failing or slow trace while storing only a few percent of the rest.”
- “All spans of a trace have to reach the same collector, so we load-balance by trace ID.”
- “Span names use the route template; concrete IDs go into attributes to keep cardinality low.”
- “We return the trace ID in error responses, so support can hand us the exact trace.”
- “We never put personal data in baggage, because it’s propagated to every downstream call.”
References
- W3C Trace Context
- OpenTelemetry: Traces
- OpenTelemetry: Sampling
- OpenTelemetry: Java agent, getting started and configuration
- OpenTelemetry: SDK environment variables
- OpenTelemetry Collector
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.