Rust Logging, Tracing and Observability: Structured Logs, Metrics and Production Diagnostics

1. Logs, traces and metrics serve different questions#

2. Add tracing#

3. Prefer structured fields#

Structured fields can be searched and aggregated more reliably than parsing free-form strings.

4. Log levels#

• ERROR : operation/system failed and needs attention.
• WARN : unusual condition or degraded behavior.
• INFO : meaningful lifecycle/business events.
• DEBUG : diagnostic detail for engineering.
• TRACE : very verbose internals.

5. Spans#

A span captures context around a unit of work so nested events inherit useful fields.

6. Request IDs#

Generate or validate a request/correlation ID at the edge, attach it to the request span, include it in downstream calls and return it to clients. This makes it possible to follow one request through many log lines.

7. Instrument functions#

Be careful not to record secrets or large request bodies automatically.

8. Metrics#

Useful service metrics often include:

• request count by route/status
• request latency histograms
• active requests
• database pool usage/wait time
• queue depth
• job success/failure count
• process memory/CPU

9. Avoid high-cardinality metric labels#

Do not use values such as user ID, request ID or raw URL as metric labels. Cardinality can explode and make the monitoring backend expensive or unstable.

10. Health vs readiness#

A health/liveness endpoint answers whether the process is alive. Readiness answers whether it is prepared to serve traffic. Readiness may include critical dependency checks, but keep checks bounded and fast.

11. Sensitive-data logging#

Never log passwords, session tokens, authorization headers, private keys or full secrets. Consider masking personal data and use allowlists for fields that may be safely recorded.

12. JSON logs#

Machine-readable logs are helpful when shipping events to centralized systems. In local development, human-readable formatting can still be useful. Make formatting configurable by environment.

13. OpenTelemetry#

For distributed tracing, tracing can be connected to OpenTelemetry-compatible exporters. This lets service spans flow into observability platforms without rewriting application instrumentation.

14. Production debugging workflow#

15. Common mistakes#

• Logging every line at INFO.
• Writing secrets into logs.
• Using only text logs with no correlation fields.
• Collecting metrics but having no alerts or dashboards.
• Instrumenting every low-level operation so traces become noise.

Mini lab#

Instrument the Rust Notes API: add request IDs, one span per request, structured completion logs, latency, database-operation spans, a /health endpoint and a /ready endpoint. Deliberately trigger a database failure and use the emitted telemetry to diagnose it.

Key takeaways#

Good observability is designed into the application. Record structured context, measure user-impacting behavior, correlate requests across layers and avoid collecting sensitive or unbounded data.