- 01Understand
- 02Prioritize
- 03Acting
Observability usually starts as a technical project and ends up as a budget line difficult to explain. Traffic grows, new labels appear and no one knows what data you actually consult. Reduce observability costs does not mean turning off signals: it means paying for useful evidence.
Where the invoice comes from
Depending on the supplier and deployment model, the cost may depend on:
- GB ingested logs, traces and profiles.
- Active metric series and cardinality.
- Hosts, containers or services observed.
- Retention and storage.
- Consultations and data analyzed.
- Transfer between regions or clouds.
- Infrastructure and staff of a self-managed stack.
The first step is to translate the invoice into technical units. Without that relationship, the equipment cannot modify its behavior.
1. Create a telemetry budget
It assigns visibility of volume and cost per service, equipment, environment and signal. Each owner should know:
- GB daily log and trace.
- Metric series active.
- Retention applied.
- Main labels by cardinality.
- Dashboards, alerts and queries that use that data.
The goal is not to collect internally from day one, but to show cause and effect.
2. Control cardinality before filtering data
A metric creates a series per tag combination. Add user_id, full URL or request ID can produce billions or millions of combinations.
High impact actions:
- Parametrize routes:
/orden/{id}. - Remove unique label IDs.
- Check histogram pockets.
- Avoid dimensions that are never grouped or filtered.
- Establish instrumentation cardinality tests.
Unique values can be stored in logs or spans if they are necessary for research.
3. Design logs by value levels
Not all logs deserve the same retention.
| Level | Example: | Strategy |
|---|---|---|
| Critical | Security, audit, transaction | High integrity and defined retention |
| Operational | Errors and Status Changes | Quick search and medium retention |
| Diagnosis | Detailed Debug | Temporary activation or short retention |
| Noise | Repeated Health Checks | Aggregation, metrics or discards |
A log that is issued thousands of times per minute and never consulted is not coverage: it is debt.
4. Apply context sampling
The head sampling decides at the start of the trace and reduces processing. The tail sampling decides when it knows the result and can prioritize:
- Mistakes.
- High latency.
- Critical services.
- Clients or strategic regions.
- Weird events.
- A sample of healthy traffic.
The policy must be tested with known incidents. A fixed percentage for all traffic is simple, but it can just eliminate the transaction that needs to be investigated.
5. Separate operational and historical retention
The last few days require quick queries. Old data can be moved to cheaper storage or added.
I defined retention by question:
- How long does it normally take to detect an incident?
- What period does audit or regulation require?
- What trend needs raw data and which one accepts aggregates?
- Is there a procedure to recover cold files?
Retaining “in case of doubt” does not replace a policy.
6. Filter near the source
Sending unnecessary data to discard them later consumes network and ingestion. OpenTelemetry Collector can filter, transform, write and sample before the backend.
However, centralized rules must be reversed and observed. An incorrect filter can hide evidence in all services at the same time.
7. Review orphan dashboards and alerts
Invent who uses each dashboard, alert and source. A resource without owner or query for months is a candidate for file or controlled deletion.
The review also reveals duplications: the same telemetry sent to two platforms during a migration can last for years.
Key metric: cost per observable service
It relates cost to service, volume and value. Some critical applications justify deep telemetry; others only require availability and errors.
A useful indicator combines:
- Monthly cost per service.
- Percentage of incidents where telemetry was used.
- Average time to find the cause.
- Volume without query.
- Change of cost by release or traffic growth.
Unit economics of telemetry
The absolute cost grows naturally with the business. To know if it is out of control, it relates consumption to a stable unit: million transactions, a thousand active users, critical service or deployment.
If the cost per million requests rises even though the stack did not change, check for new labels, repetitive logs or lower sampling efficiency. If you go down, confirm that coverage and diagnostic time did not get worse.
This metric allows you to talk with Finance without reducing the discussion to “the tool is expensive.” It also helps project growth scenarios and negotiate commitments based on real volume.
30-day reduction plan
- Days 1–5: Map collection units and top consumers.
- Days 6–10: correct extreme cardinality.
- Days 11–15: remove repetitive logs and permanent debug.
- Days 16–20: design criticality sampling and retention.
- Days 21–25: archive dashboards and pipelines without owner.
- Days 26–30: establish budgets and consumer alerts.
Each change must be compared against a diagnostic test to verify that no essential evidence is lost.
Save a baseline before optimizing: volume, cost, consultation time and ability to resolve three known incidents. I repeated the same investigations later. Savings are valid only if the evidence remains available within the time the operation needs.
Next step
CTL can make an assessment of costs, cardinality and coverage on commercial platforms or open source. Set up a review to identify savings without compromising SLOs or responding to incidents.
Frequently Asked Questions
Does Open source eliminate the cost?
Removes or reduces licenses, but maintains computation, storage, network, high availability, upgrades and specialist hours. Total cost should be compared.
Should I delete health checks logs?
They can often become metrics or sample, but first they must be checked for audit or diagnostic function.
Who should govern the cost?
A platform or observability practice can define standards, but service-owner teams must see and control their consumption.