Phoenix Observability with Elixir Telemetry: Metrics, Prometheus/Grafana Dashboards, and OpenTelemetry Tracing

Your Phoenix app is sprinting across nodes, juggling requests, juggling database calls, and juggling background jobs. It is a circus act that deserves a spotlight, not guesswork from the bleachers. That is where Elixir Telemetry steps in. It gives you clear, structured signals from your application, so you can find the hot path, the slow path, and the oh-no-why-is-that-on-fire path. 

If you care about reliability, performance, and, frankly, keeping your evenings free, this lightweight instrumentation approach will become your favorite backstage pass. In the world of software development, observability turns a chaotic production system into a readable story, one event at a time.

Why Observability Matters in Distributed Phoenix Apps

Distributed Phoenix apps are powerful because the BEAM loves concurrency the way a barista loves espresso. Processes are cheap, nodes can mesh, and throughput scales without drama. That flexibility can hide failures behind busy traffic, so incidents start as whispers. A login endpoint grows sluggish only when a certain header is present. A multi-tenant query explodes when one outlier account adds a thousand filters. 

Without observability, these patterns look like random gremlins. Observability does not just help you fix problems. It helps you avoid them. When you can see latency distributions, not only averages, you can tune your pool sizes and cache policies before users feel pain. When you can trace a single request across services, you trim time-to-resolution from hours to minutes. Telemetry is the connective tissue that turns all these signals into a coherent body of insight.

How Telemetry Works in Elixir

Telemetry is an event system. Libraries emit events with a concise name, a small map of measurements, and a metadata map that adds context. You attach handlers that listen for those events, then push them to metrics, logs, or tracing. The design is simple on purpose. It keeps overhead small in production and keeps library authors honest about what they expose.

Handlers act like translators. They turn low-level events into the metrics and spans that your tools understand. Because handlers are just functions, you can add, remove, or tweak them without changing the libraries that fire the events. Your app stays decoupled, and the code that actually does work is not entangled with your code that observes work.

Events, Metadata, and Handlers

An event might be named [:phoenix, :endpoint, :stop]. The measurements could include the total duration, and the metadata could identify the connection or the route. Your handler receives these values, decides which labels to keep, and records the measurements in your chosen backend. Good handlers are conservative about labels to avoid metric-cardinality explosions, and careful about converting units so dashboards stay consistent.

Metrics and Pollers

Not every signal is event based. Some stats are best pulled on an interval, like VM memory, scheduler utilization, or ETS table sizes. That is where pollers fit. They run on a timer, emit periodic events, and feed the same handler pipeline. The result is a single, unified way to handle push and pull style signals.

Phoenix, Ecto, and The Built-In Signals

Phoenix and Ecto already speak Telemetry. Routers emit start and stop events around requests, channels report join and handle timings, and Ecto instruments query planning and execution. You get timings for endpoints, status codes, and database call durations with minimal work. This is the foundation for latency histograms, error rates, and p95 goals that are grounded in reality.

Because these signals arrive with rich metadata, you can label metrics with route names, HTTP methods, and repo names without custom patches. That makes it easy to drill from a global latency chart to a single troublesome action. Keep labels stable across releases, and your graphs will not reset every deploy.

Tracing Across Services With OpenTelemetry

Metrics tell you there is a problem. Tracing shows where it hides. OpenTelemetry integrates with Elixir Telemetry so you can connect spans across processes, nodes, and even languages. A user request that starts in Phoenix can follow an Ecto query, hop through a Redis cache, and end in a Go service. When the trace lights up a slow step, you can focus your energy precisely where it counts.

In Elixir, spans are cheap and composable. You wrap critical sections of code with spans and pass context through headers on outbound calls. Libraries and middleware can enrich spans with attributes like route names or SQL operation types. Keep span names readable, add only the attributes that matter, and avoid dumping raw payloads. You will get clean timelines that clarify, not confuse.

Metrics Pipelines That Scale

You need a pipeline that drinks from the firehose without choking. Prometheus is a solid default for scraping metrics and powering Grafana. StatsD can work if your infrastructure prefers that route. Either way, your handlers should consistently convert durations to seconds, sizes to bytes, and counters to integers. Precision is not a luxury here. It is how your dashboards maintain your trust.

Histograms are your best friend for latency. Averages can hide tail pain that wakes up on-call engineers at 3 a.m. Use bucket boundaries that reflect your app. A millisecond-obsessed service needs fine buckets below 50 ms, while a job queue that runs longer tasks needs wider buckets at the top end. Tune these once, verify with production distributions, then stick with them so month-over-month trends are meaningful.

Designing Events That Age Well

Well designed events are short, stable, and focused. Choose names that mirror your domain. If your app runs tenants and invoices, consider events like [:billing, :invoice, :finalize, :stop] with measurements for duration and metadata for tenant and invoice type. Avoid attaching entire structs or user identifiers. That is a tangled path to privacy violations and metric bloat.

Cardinality is the silent budget of observability. Labels that multiply unboundedly, like unique IDs or raw URLs, will explode time series counts. Prefer normalized route names, capped enumeration values, and a small set of tenants if you must label by tenant. The right choice here keeps storage affordable and queries fast, which means dashboards load when you need them most.

Dashboards, Alerts, and Sanity Checks

A good dashboard feels like a cockpit. One panel shows global request rate and error rate. Another shows latency percentiles with clear thresholds. A third breaks down database timings and cache hit rates. When an alert fires, you want to land on a dashboard that already answers the first three questions in your head. What changed, how bad is it, and which component looks guilty.

Alerts should be specific. Alert on error ratio, not raw error count. Alert on p95, not average. Alert on sustained thresholds, not single spikes. Add a little hysteresis to avoid paging for brief blips. The goal is confidence, not noise. If every alert feels like a false positive, your team will quietly mute the channel and hope for the best. That is not a strategy. The same tradeoff shows up well beyond infra metrics: teams running data drift detection pipelines for production ML models face an identical balance between catching real signal and drowning in noise.

Testing and Local Feedback Loops

Treat Telemetry like any other contract. Write tests that assert your events fire with the right names and fields. In local development, print selected events at a low volume to confirm your mental model matches reality. You can even record short traces during feature work, then open them in a trace viewer to ensure context flows end to end. That habit pays off when you land a complicated change and want production to behave.

Do not forget to test your fallback paths. Timeouts, cancellations, and retries should emit events that make sense. If your code short circuits on an error, the Telemetry story should not go blank right when you need it most. A little extra attention here can turn a baffling outage into a clear narrative.

Operational Safety and Cost Awareness

Observability is not free. Every metric, trace, and log eats a little CPU and a little budget. Keep sampling in your toolbox. Sample traces aggressively on high volume endpoints, then crank the rate to 100 percent when you are chasing a bug. Store only the labels you query. Archive old metrics or downsample them after the period when you actually need full fidelity.

Security and privacy matter as much as performance. Scrub secrets from metadata, and never attach raw tokens or PII to events. Treat your dashboards as windows into production, not a suitcase for sensitive data. A clean Telemetry design lets you share dashboards with stakeholders confidently, which is great for alignment and terrible for mystery.

Conclusion

Telemetry gives Phoenix apps a calm, reliable voice. Instead of wrestling with logs that read like cryptic haikus, you get structured events, sturdy metrics, and clean traces. The BEAM makes concurrency friendly, and Telemetry makes the results understandable. Build with clear event names, measured labels, and consistent units. Add tracing where humans get lost. Keep dashboards tidy, alerts honest, and sampling sensible. 

The payoff is faster incident response, smarter tuning, and a system that feels less like a haunted house and more like a well lit studio. Your future self, and your sleep schedule, will be grateful.