Service Discovery & Health Probing in Kubernetes & Consul: Preventing Cascading Outages

In cloud-native container platforms where pods are constantly created, rescheduled, and destroyed across nodes, static IP addresses no longer exist. Automated service discovery combined with resilient health probing is essential to ensure traffic routes strictly to healthy instances.

1. The Three Kubernetes Health Probes

Startup Probe: Guards slow-starting applications (e.g. warming local caches or loading ML models) from premature restarts.
Liveness Probe: Detects unrecoverable deadlocks. Failure causes Kubernetes to kill and restart the container.
Readiness Probe: Checks if the container is ready to accept incoming traffic. Failure simply removes the pod from Service endpoints without restarting it.

Critical Reliability Warning

Never ping external dependencies like PostgreSQL inside a Liveness Probe! If the database experiences a momentary slowdown, all cluster pods fail liveness simultaneously, triggering a catastrophic cluster-wide thundering herd restart!