Reading mapFrom the signal to an operational decision
  1. 01Understand
  2. 02Prioritize
  3. 03Acting

Kubernetes automates deployments and recovery, but also multiplies ephemeral components, labels and relationships. A pod that disappears can take the evidence of the problem; a “green” cluster can host a slow application. Kubernetes observability connect platform and experience, not just counting pods.

The Five Layers to Observe

1. User experience

Availability, errors and latency of critical transactions. It is the layer that determines impact. It can be measured with RUM, synthetics and application metrics.

2. Applications and services

Rate, errors, duration, dependencies and traces. Must include deployed version, namespace, service and owner.

3. Workloads

Deployments, StatefulSets, DaemonSets, Jobs and Pods. Restarts, pending states, image errors, tests and limits matter.

4. Nodes and resources

CPU, memory, disk, network, pressure, assignable capacity and saturation. You must distinguish real use of configured requirements and limits.

5. Control plane

API server, scheduler, controllers and etcd. Its degradation can affect changes, scaling and recovery even if applications continue to respond for a while.

Metrics: start with symptoms

Kubernetes metrics are exposed from components, kubelet and tools such as kube-state-metrics. The risk is to turn each metric into an alert.

Prioritize:

  • Latency and critical service errors.
  • Pods not available regarding desired status.
  • Persistent restarts and CrashLoopBackOff.
  • Workloads pending due to lack of resources or restrictions.
  • Node saturation and pressure.
  • Errors and latency of the API server.
  • Health of etcd and storage latency.
  • Loss or delay of telemetry pipeline.

An alert must indicate near impact or risk and be responsible. The rest may be available for diagnosis.

Logs: evidence disappears with pod

Containers usually write to stdout y stderr; the runtime and kubelet manage files in the node. Kubernetes does not offer long-lasting central storage. An agent or collector is needed to forward the logs before the node rotates or disappears.

Good practices:

  • Logs structured in JSON.
  • Timestamp and consistent severity.
  • service.name, environment, version, namespace and pod.
  • trace_id y span_id For correlation.
  • Writing of secrets and personal data.
  • Volume and retention limits by category.

Do not use pod names as your main identity: they change. Identity must represent the service and version.

Traces: follow a request between pods

The distributed traces show how a request goes through inputs, services, queues and databases. OpenTelemetry can instrument applications and send OTLP to a Collector deployed in the cluster.

A useful trace allows you to answer:

  • Where was latency added?
  • What dependency failed?
  • Does the failure affect a version or zone?
  • Are there retryings that amplify charge?
  • What are the logs for the transaction?

Without context propagation, each microservice appears as an island.

Collection architecture

A common pattern combines:

  • DaemonSet for node telemetry, local log and metrics.
  • Gateway Collector for processing, sampling and exportation.
  • Prometheus for metric scraping.
  • Logs and trace backend, managed or self-managed.

The design should include peaks, backpressure, retrying, memory and behavior when the backend is not available. Observability should not knock down the application you are trying to observe.

Cardinality: the hidden cost of labels

Kubernetes adds useful dimensions — cluster, namespace, pod, container — but combining them with unique IDs can multiply series.

  • Pod and container UID.
  • Unparametrized HTTP paths.
  • User IDs, request or session.
  • Labels created dynamically.
  • Histograms with too many buckets.

Keep details in logs or traces when not required as a metric label.

Minimum Kubernetes Dashboard

An operating dashboard should show:

  1. SLO and RED signals for critical applications.
  2. Desired vs workloads status available.
  3. You could go through, failed and restarted.
  4. Node and namespace capacity and pressure.
  5. Errors and latency of control plane.
  6. Recent changes and deployments.
  7. Health and loss of telemetry pipeline.

It will separate the executive view from the platform view. A manager needs impact and trend; an SRE needs detail to act.

Multi-cluster: preserving context without multiplying boards

When multiple clusters exist, copying the same dashboard for each one makes it difficult to compare and adds maintenance. It will standardize attributes such as organization, account, region, cluster, environment and equipment. Then I allowed to filter from a global view into detail.

Do not centralize everything without evaluating connectivity and isolation. A regional pipeline can reduce transfer and maintain operation when the central link fails. For regulated services, the log and trace residence may require different backends or retentions.

I also measured the health by cluster of the Collector, export delays and discarded volume. If a region stops sending telemetry, a global view could interpret the absence of data as absence of incidents.

Frequent errors

  • Alert for each restart without considering the final state.
  • Observe nodes but no transactions.
  • Keep logs without limits.
  • Use imported dashboards without adapting them to SLOs.
  • Do not label responsible.
  • Ignore the cost of cardinality.
  • Do not prove what happens when the Collector fails.

Before expanding the stack, run a test: remove a pod, satulate a dependency in a controlled environment and simulate unavailability of the backend. The team should identify impact, service and cause without depending on the name of the pod that has already disappeared. Register missing time, steps and telemetry; that evidence defines the following improvement much better than installing another dashboard.

Next step

CTL integrates Kubernetes with Prometheus, Grafana y OpenTelemetry. Provide an evaluation to review coverage, alerts and cost.

Frequently Asked Questions

What is the difference between metrics-server and Prometheus?

Metrics Server delivers metrics of resources for autoscaling and autoscaling functions kubectl top; not intended as a complete historical solution. Prometheus collects and stores series for analysis and alerts.

Is a cluster collector convenient?

Depends on volume, isolation and availability. A cluster gateway simplifies context and faults, while regional gateways can centralize processing. Architecture must be tested under load.

How do you see a pod that's already gone?

Telemetry should be exported outside the node and associated with stable identities. Kubernetes events, centralized logs and traces retain evidence after completion of the pod.

Verifiable sources