⏱️ Lectura: 14 min
Netflix shuts down production servers on purpose, every day, and that’s how it builds the most reliable streaming service in the world. The practice is called chaos engineering, and it consists of injecting controlled failures into real systems to discover their weak points before an outage, a traffic spike, or a full disk finds them for you.
📑 En este artículo
- TL;DR
- What Chaos Engineering Is and Why It Matters
- How a Chaos Experiment Works, Step by Step
- Practical Examples: From a Simple Script to a Kubernetes Experiment
- How to Start Today: Installing Chaos Mesh on a Test Cluster
- Real-World Use Cases
- Common Mistakes and Best Practices
- Comparing Chaos Engineering Tools
- Going Deeper: Game Days and Expanding the Blast Radius
- Frequently Asked Questions
- References
It’s not about running random scripts or breaking things for the sake of it: it’s a method with hypotheses, metrics, and a blast radius measured in advance. In this article, you’ll see how it works, which tools to use, and how to set up your first experiment today.
TL;DR
- You’ll understand the four formal principles of chaos engineering and where they come from.
- You’ll install Chaos Mesh on Kubernetes with Helm and launch your first PodChaos experiment.
- You’ll learn when to choose Chaos Monkey, Chaos Mesh, Litmus, or AWS Fault Injection Simulator.
- You’ll be able to define a safe blast radius before running an experiment in production.
- You’ll know how to check with kubectl whether a chaos experiment is still active or has finished.
- You’ll identify the most common mistakes when starting out, like not having a panic button.
What Chaos Engineering Is and Why It Matters
Chaos engineering is the discipline of experimenting on a distributed system to build real confidence in its ability to withstand turbulent conditions in production. The formal definition comes from the Principles of Chaos Engineering manifesto, written by Netflix engineers who created the first tool of its kind, Chaos Monkey, in the early 2010s.
The idea was born from a concrete problem. Netflix was migrating from its own data centers to AWS and needed a guarantee: if an instance died without warning, something that happens all the time in the cloud, the service couldn’t go down with it. Instead of waiting for something to fail on its own, the team wrote a program that killed random instances during business hours, forcing every team to design fault-tolerant services from the start.
Today the concept has expanded well beyond killing a machine. Teams inject network latency, fill up a disk, saturate the CPU, cut off access to a database, or simulate the failure of an entire availability zone. The goal is always the same: find the weakness in a controlled environment, with people watching the metrics dashboard, instead of discovering it on a weekend with the entire system down.
This is what sets it apart from a classic disaster recovery drill. A drill is usually a planned event that happens once a year; a chaos engineering experiment is meant to run continuously and automatically, as just another test in the pipeline, not as a special occasion.
How a Chaos Experiment Works, Step by Step
Every serious experiment follows the same cycle, regardless of which tool you use. First you define the steady state: a business metric, not a purely technical one, that represents the system working well, for example p99 latency or the rate of completed checkouts per minute. Then you formulate a hypothesis: if we kill a pod in the payments service, the steady state holds because replicas and retries exist.
With a clear hypothesis, you inject the real failure and compare what you observe against what you expected. If the steady state holds, the hypothesis is confirmed and you gain verifiable, not theoretical, confidence. If the steady state breaks, you found a real weakness before a customer did.
flowchart TD
A["Define steady state"] --> B["Formulate a hypothesis"]
B --> C["Inject a controlled failure"]
C --> D["Observe live metrics"]
D --> E{"Does the hypothesis hold"}
E -->|"Yes"| F["Confirm the system is resilient"]
E -->|"No"| G["Fix the weakness found"]
G --> A
F --> AThe Four Formal Principles
The first principle calls for defining the steady state in measurable terms, not assumptions. The second calls for varying real-world events: hardware failures, network faults, traffic spikes, and third-party errors, not just shutting down a server.
The third principle, the most resisted in practice, calls for running experiments as close to production as possible, because a staging environment almost never reproduces real traffic and dependencies. The fourth calls for automating experiments so they run continuously, instead of being executed once and shelved.
Observability Comes Before Chaos
No chaos experiment makes sense if nobody can see what happened during the failure injection. Before installing any chaos tool, it’s best to have metrics (Prometheus), traces (OpenTelemetry), and centralized logs already running, because the experiment only delivers value if you can compare the steady state before, during, and after the failure.
Practical Examples: From a Simple Script to a Kubernetes Experiment
The simplest possible chaos experiment is a script that kills random processes. It doesn’t use any specialized tool, but it illustrates the core idea: introducing a real failure in a repeatable way.
#!/bin/bash
# hello-world-chaos.sh: kills a checkout-service process every 60 seconds
while true; do
PID=$(pgrep -f checkout-service | shuf -n 1)
if [ -n "$PID" ]; then
kill -9 "$PID"
echo "Process $PID killed at $(date)"
fi
sleep 60
doneWhen you run this script against a checkout-service instance with multiple replicas, you should see in the logs how the process dies and the load balancer redirects traffic to another replica without the user noticing anything. If the user does notice an error, you just found your first real weakness.
In Kubernetes, the production-grade equivalent is a Chaos Mesh manifest, which declares the experiment as just another resource in the cluster:
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
name: kill-pod-checkout
namespace: chaos-mesh
spec:
action: pod-kill
mode: one
selector:
namespaces:
- produccion
labelSelectors:
app: checkout-service
scheduler:
cron: "@every 10m"This manifest tells the Chaos Mesh operator to pick a pod with the label app: checkout-service in the produccion namespace every 10 minutes and kill it. The difference from the bash script is huge: here the experiment is version-controlled in Git, has a declarative scheduler, and can be reverted with a single kubectl delete.
⚠️ Heads up: never run your first chaos experiment directly in production without notifying the on-call team. The goal is to learn, not to cause a real incident without active monitoring.
Injecting Network Latency Instead of Killing the Pod
Killing a pod is the simplest failure, but often the real problem is subtler: a dependency that responds slowly. Chaos Mesh also lets you inject latency with a NetworkChaos resource:
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: latencia-checkout
namespace: chaos-mesh
spec:
action: delay
mode: all
selector:
namespaces:
- produccion
labelSelectors:
app: checkout-service
delay:
latency: "300ms"
jitter: "50ms"
duration: "5m"This experiment adds 300 ms of latency, plus 50 ms of jitter, for 5 minutes, to all of the service’s traffic. It’s the kind of failure that tends to reveal misconfigured timeouts that a simple pod-kill never exposes.
sequenceDiagram
participant Op as Chaos Mesh Operator
participant K as Kubernetes API
participant Pod as Target Pod
participant Mon as Prometheus
Op->>K: applies PodChaos manifest
K->>Pod: injects the declared failure
Pod-->>Mon: exposes degraded metrics
Mon-->>Op: confirms the experiment is active
Note over Op,Pod: the experiment reverts automatically when duration expiresHow to Start Today: Installing Chaos Mesh on a Test Cluster
To try the examples above on your own cluster, minikube or kind work fine if you don’t have one handy. Install Chaos Mesh with Helm:
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm repo update
kubectl create ns chaos-mesh
helm install chaos-mesh chaos-mesh/chaos-mesh \
--namespace=chaos-mesh \
--set chaosDaemon.runtime=containerd \
--set chaosDaemon.socketPath=/run/containerd/containerd.sockWith the operator installed, apply the PodChaos manifest from the earlier example:
kubectl apply -f kill-pod-checkout.yaml
kubectl get podchaos -n chaos-meshTo confirm the experiment is actually running, not just declared, check its status:
kubectl describe podchaos kill-pod-checkout -n chaos-meshLook for the Phase field in the output: Running confirms Chaos Mesh is injecting the failure at that moment; Waiting means it’s waiting for the next cron trigger. Chaos Mesh also exposes a web dashboard that keeps the history of every experiment and its result.
Real-World Use Cases
For years, Netflix maintained what it called the Simian Army, a family of tools derived from Chaos Monkey: Latency Monkey to inject artificial delays, Conformity Monkey to detect instances that didn’t follow best practices, and Chaos Kong to simulate the failure of an entire AWS region. The origin and the code are still documented in the public Chaos Monkey repository.
AWS formalized the approach as a managed product with AWS Fault Injection Simulator, which lets you define experiments (killing EC2 instances, saturating the CPU, cutting off access to an API) without installing anything on your own cluster. It’s the simplest path for teams already living inside the AWS ecosystem.
In the open source world, teams running Kubernetes outside a managed cloud usually choose between Chaos Mesh and Litmus Chaos, both projects under the Cloud Native Computing Foundation. The typical choice depends on whether the team already thinks in CRDs as its main configuration approach, where Chaos Mesh feels more native, or whether it prefers a catalog of reusable experiments, the strong point of Litmus’s ChaosHub.
E-commerce platforms use these experiments before high-traffic events to verify that the payments system can withstand the failure of an external provider, and banks and fintechs apply them to test that a transaction retries or rolls back correctly if the fraud service doesn’t respond in time.
Common Mistakes and Best Practices
- Starting without limiting the blast radius: your first experiment should never be able to affect more than 1% of traffic. Adjust the
selectorso it targets a single replica or a canary, not the entire deployment. - Not having a panic button: every experiment needs a way to abort it within seconds. In Chaos Mesh,
kubectl delete podchaos kill-pod-checkout -n chaos-meshdoes the job; if a homemade script doesn’t have an equivalent, better not to run it. - Confusing load with chaos: a load test measures how much traffic the system can handle; a chaos experiment measures what happens when something breaks under normal traffic. They’re complementary, not the same thing.
- Not automating: a manual experiment that runs once and gets shelved doesn’t build continuous confidence. The manifesto’s fourth principle calls precisely for automating experiments so they run on their own, all the time.
- Skipping the game day: before automating, it’s worth running the experiment live with the whole team watching the metrics dashboard, so you can abort and learn together.
💡 Tip: always start in an environment that’s a faithful replica of production before moving the experiment to the real thing. If that environment doesn’t reflect production, the experiment tells you nothing useful.
None of these mistakes is purely technical. Most of them show up when chaos engineering is treated as just another infrastructure tool instead of a practice that needs support from the whole team, including whoever’s on call and whoever defines which business metric actually matters.
Comparing Chaos Engineering Tools
| Tool | When to Use It | Advantage | Limitation |
|---|---|---|---|
| Chaos Monkey | AWS instances with Spinnaker already integrated | Pioneer, tested for more than a decade at Netflix | Built for instances, not designed for Kubernetes |
| Chaos Mesh | Self-managed Kubernetes clusters, on-premise or in the cloud | Experiments as native CRDs, with a built-in dashboard | Requires operating your own cluster and its RBAC |
| Litmus Chaos | Teams that want a catalog of reusable experiments | ChaosHub with pre-packaged experiments | Steeper learning curve for simple cases |
| AWS Fault Injection Simulator | Infrastructure already centralized on AWS | Managed service, no need to operate your own infrastructure | Tied to the AWS ecosystem |
No option is universally better: the decision depends on where the infrastructure already lives and how much declarative control the team needs over each experiment.
Going Deeper: Game Days and Expanding the Blast Radius
Mature teams don’t run experiments directly against 100% of production. They scale the blast radius in stages: first against development, then in staging, then against a canary that receives a minimal fraction of real traffic, and only at the end against full production, with the experiment already proven in the earlier stages.
flowchart LR
A["Development environment"] --> B["Staging"]
B --> C["Canary in production, minimal traffic"]
C --> D["Full production"]
subgraph "Expanding blast radius"
A
B
C
D
end
This escalation coexists with the game day practice: a scheduled session where several teams, not just infrastructure, watch a live experiment together. The advantage over automating it right away is human: it builds shared context about how the real system reacts, and that knowledge later translates into better-calibrated runbooks and alerts.
A detail that’s often overlooked is that the steady state isn’t a single technical metric but a set agreed upon with the product team. Netflix, for example, didn’t measure only whether a server responded, but whether the user could keep playing a video. That distinction between a technical metric and a business metric separates a useful experiment from one that passes on the technical dashboard but still leaves the user staring at a blank screen.
A team’s typical maturity progression goes through three stages: manual, ad hoc experiments; scheduled experiments managed with an internal catalog; and finally automated experiments that run on their own inside the deployment pipeline, without human intervention on each run.
💭 Key takeaway: the ultimate goal of chaos engineering isn’t to prove the system is perfect, but to find the next weakness before a real incident finds it for you.
📖 Summary on Telegram: View summary
Your next step: install Chaos Mesh on a minikube cluster and run this article’s PodChaos manifest against your own deployment with at least two replicas.
Frequently Asked Questions
Is chaos engineering the same as breaking things at random?
No. Every experiment starts from a clear hypothesis about the system’s steady state and is measured against that hypothesis. Breaking something without measuring anything doesn’t generate any learning.
Do you need Kubernetes to practice chaos engineering?
No. Chaos Monkey was born for AWS instances without Kubernetes, and AWS Fault Injection Simulator works directly on EC2, RDS, or Lambda without a cluster in between.
Is it safe to run experiments directly in production?
It’s safe if the blast radius is limited, a panic button exists, and the on-call team knows the experiment is running. Without those three conditions, it’s not advisable.
What is a game day?
A scheduled session where several teams watch a chaos experiment live, with the goal of building shared context and spotting gaps in alerts and runbooks.
How much does it cost to get started?
Chaos Mesh and Litmus Chaos are open source and free; the only cost is the test infrastructure where they run. AWS Fault Injection Simulator charges per experiment executed inside Amazon’s cloud.
Does chaos engineering replace traditional testing?
No. Unit and integration tests verify that the code does what’s expected under normal conditions; chaos engineering verifies that the entire system, with all its real dependencies, keeps working when something fails.
References
- Principles of Chaos Engineering: the original manifesto with the four formal principles.
- Netflix/chaosmonkey on GitHub: the source code of the tool that started the discipline.
- Chaos Mesh: official documentation for the Cloud Native Computing Foundation’s Kubernetes project.
- Litmus Chaos: the project’s official site and its ChaosHub catalog.
- AWS Fault Injection Simulator: official page for AWS’s managed service.
📱 Enjoy this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.
0 Comments