Quiet changes lead to loud incidents

A diff tells you what changed. It does not tell you where the consequences stop.

A route weight flips from v1 to v2. Cache hit rate slips. Retries kick in. A shared replica hits its connection cap. Checkout starts timing out. Nobody shipped a “big” change. That is the point.

In production, small usually describes the diff, not the risk. What matters is change risk: the gap between the local intent of a change and the system behavior that appears when that change meets real traffic, real dependencies, and real limits. A three-line change can still become a system event.

That is why so many ugly incidents begin with something ordinary: a config edit, a routing change, a dependency bump, a timeout tweak, an IAM rule, or a Terraform plan that looked routine in review. The code can be correct. The outcome can still be bad.

The mental model: diff, reach, amplifiers, limits

A useful way to reason about these changes is with four questions.

Diff: What actually changed?

Reach: Which services, stores, queues, network paths, or trust boundaries can feel that change downstream?

Amplifiers: What can multiply the effect? Retries, fan-out, cache invalidation, autoscaling lag, failover, queue buildup, or shared dependencies.

Limits: Where are the hard edges? Timeouts, connection caps, quotas, p99 budgets, shard hot spots, rate limits, or auth boundaries.

The naive intuition is that a small diff should have a small blast radius. That only works when reach is narrow, amplifiers are weak, and limits are far away. Production often gives you the opposite mix.

A change becomes dangerous when it is locally simple and systemically connected.

A realistic failure chain

Consider an ecommerce checkout system doing about 1,200 requests per second during a promotion. checkout-api calls pricing. The team makes what looks like a modest service-mesh change: route 100 percent of traffic to pricing-v2.

Nothing about the review screams danger. Same API. Same owner. No schema change. Staging is green. A small canary also looks fine.

Then production adds the parts that the diff did not contain. pricing-v2 normalizes cache keys a little differently. Cache hit rate drops from 97 percent to 82 percent. Median latency rises from 35 ms to 95 ms. P99 moves from 180 ms to 650 ms.

That would be survivable in a linear system. It is not one. checkout-api times out downstream calls at 300 ms and retries twice. Once enough traffic crosses that threshold, effective load on pricing more than doubles. The read replica behind pricing climbs from about 400 active connections to 980 against a cap of 1,000. Connection acquisition starts queueing. A background cache warmer falls behind. Miss rate gets worse. Two other services sharing that replica start showing latency spikes even though nobody changed them.

The autoscaler adds pods after about 90 seconds, but CPU was never the real bottleneck. Connection wait was. More pods add more contention. Error rate rises from 0.2 percent to 18 percent. Checkout starts failing.

Those numbers are illustrative, but the shape is common. The outage does not begin with “the database is down.” It begins with a local change whose reach was wider than expected, whose amplifiers were already sitting there, and whose limits were closer than anyone thought.

Smart teams still miss this

Good teams are not careless. They are usually making decisions with incomplete visibility.

Staging is useful. It tells you whether a change can work in a controlled environment. It does not reliably tell you how that change will behave against the current production graph, production data shape, and production load.

Observability is essential. It tells you what is happening and what just broke. It is strongest after the change is live. It does not answer the pre-deploy question: what is this likely to affect?

Static scanning and policy checks are useful too. They catch known bad patterns. They are much weaker at novel combinations of otherwise-valid changes and live dependencies.

So teams fill the gap with experience, rollout caution, architecture diagrams, and a growing list of folklore. That is not irrational. It is what people do when the diff is easy to see and the runtime reach is not.

The missing step

This is where the argument needs to stay honest.

Pre-production change evaluation is not a claim that you can predict every emergent failure in a distributed system. You cannot.

It is not a replacement for canaries, load tests, observability, or human judgment. You still want all of those.

It is a narrower and more practical step: evaluate a proposed change against a current model of the environment before it ships, and ask what the change can reach, which bad paths it could open, and which limits are already close.

Sometimes that should lead to a hard stop. Often it should lead to something simpler: a slower rollout, a more targeted canary, a different retry policy, an extra reviewer, or just a more informed conversation before deploy.

The same idea can help with security too. A network rule or IAM change may look local in review while still altering a data path or widening a trust boundary in production. That does not replace dedicated security review. It does make the review more concrete.

For leaders, the tradeoff is not “more process versus less process.” It is a little more analysis on the small set of risky changes versus a lot more surprise on the back end. The goal is not to slow every deploy. It is to stop treating all diffs as equally legible.

Where Stag fits

At Stag, this is the gap we are focused on.

Our approach is to model live infrastructure and proposed changes together. In practice, that means pulling in cloud and Kubernetes state alongside Terraform plans, computing the forward reach of a change — the downstream services, stores, and paths it can affect — and surfacing inspection or simulation results in CI and the product UI before rollout.

The claim is intentionally modest. Not that a model can predict everything. That would be nonsense. The useful claim is that teams should know more about likely production impact before production becomes the thing doing the evaluation.

A practical rule

For changes that touch routing, retries, timeouts, dependencies, data paths, IAM, network policy, or shared infrastructure, ask four questions before deploy:

1. What changed?
2. What can it reach?
3. What can amplify it?
4. Which limits are close enough to matter?

If your team can answer those questions from current system data, good. If not, the change is less understood than it looks.

That does not mean every diff needs a tribunal. A small, isolated, low-criticality system with easy rollback and little shared infrastructure probably does not need much more than good engineering judgment. Customer-facing paths, shared services, and infrastructure controls are a different category. Treating them that way is not bureaucracy. It is accuracy.

Conclusion

The expensive part of many outages is not that the original change was reckless. It is that the change looked local when it was not.

A diff is a starting point. It is not a blast-radius model.

The recommendation is straightforward: for changes that affect shared runtime behavior, add a pre-production change evaluation step. Not because it will predict everything. Because it can catch more than a code review, earlier than observability, and with less guesswork than folklore.

And when is that extra step unnecessary? When the system is truly small, isolated, cheap to fail, and easy to reverse. Many production systems stop meeting those conditions earlier than teams admit.