What would you do if your most important dependency disappeared for six hours? Contingency planning is the engineering discipline of deciding, before the incident, which functions must survive, how degraded they may become, and what people and systems will do next. You will leave with a compact method for turning vague fears into tested recovery choices.
Imagine a payment service whose database region has failed. The team has backups, a second region, and a detailed restore script. Yet after three hours, nothing useful is back: engineers restored dashboards first because they were easiest, the payment provider's credentials are in the unavailable secret store, and nobody knows who can authorize a degraded mode.
That is the characteristic failure of a plan that lists components instead of preserving a function. The question is not “How do we restore every server?” It is:
The second choice is the heart of contingency planning. Disruption creates scarcity: time, staff, information, power, suppliers, and safe operating capacity are all limited. A plan is valuable when it makes the trade-off explicit before the trade-off becomes a fight.
A hazard is only interesting because of what it prevents you from doing. A region outage, ransomware event, flood, or key supplier failure can all produce the same operational question: which capability is now missing, and how long can the organization tolerate that?
Use a business impact analysis (BIA) to answer that question. For each function, record the consequence of delay, the latest acceptable restoration time, the tolerable data loss, and the resources that make the function possible. NIST describes BIA as a way to determine criticality, impact, and recovery time objectives; FEMA similarly frames it around the consequences of failing to perform essential functions.
Recovery Time Objective: how long the function may be unavailable before the target is missed.
Recovery Point Objective: how much recent data loss is acceptable, measured backward in time.
Maximum Tolerable Downtime: the outer limit after which the consequences become unacceptable.
These are not magical constants. They are negotiated consequences. A 15-minute RTO for payments might be justified by lost revenue and trust; a 24-hour RTO for analytics might be entirely reasonable. The point is to spend resilience budget where delay hurts.
| Function | RTO | RPO | Acceptable degraded mode |
|---|---|---|---|
| Accept payment | 15 min | Near zero | Queue signed requests; disable nonessential fraud enrichment |
| Issue invoice | 4 h | 1 h | Batch generation when the ledger is healthy |
| Show analytics | 24 h | 24 h | Serve yesterday’s cached snapshot |
Once an incident starts, recovery is an allocation problem. Let each function have an impact rate \(c_i\) while unavailable, a restoration cost \(t_i\), and a deadline \(d_i\). With a recovery budget \(B\), the plan chooses actions that reduce the most consequential loss before deadlines:
You do not need to solve this as a perfect optimization model. The equation exposes the hidden question: if you can only restore two things, which two preserve the mission? Priority is a policy decision informed by the BIA, not an accidental consequence of which runbook is shortest.
python3 prioritized-recovery.py)# Run: python3 prioritized-recovery.py — proves scarce recovery capacity must follow service priority.
services = {
"payments": {"time": 50, "essential": True},
"invoices": {"time": 30, "essential": False},
"dashboards": {"time": 40, "essential": False},
}
budget = 90
def recover(order):
used, restored = 0, []
for name in order:
if used + services[name]["time"] <= budget:
restored.append(name)
used += services[name]["time"]
return restored, used
# The order is the policy: build order accidentally restores the wrong promise.
for label, order in [
("naive build order", ["dashboards", "invoices", "payments"]),
("priority order", ["payments", "invoices", "dashboards"]),
]:
restored, used = recover(order)
status = "UP" if "payments" in restored else "DOWN"
print(f"{label:18} -> {', '.join(restored):20} ({used:2} min), payments: {status}")
$ python3 prioritized-recovery.py
naive build order -> dashboards, invoices (70 min), payments: DOWN
priority order -> payments, invoices (80 min), payments: UPThe previous sections plan recovery at the organizational level. Contingency model predictive control (CMPC) applies the same idea to a dynamical system: while the controller follows a high-performance nominal trajectory, it also preserves a feasible alternate trajectory for a known possible emergency.
Why is that different from ordinary MPC? Ordinary MPC asks, “What is the best legal trajectory under the model I currently believe?” If a hazard appears later—ice, an obstacle, loss of an actuator, or a sudden change in friction—the best nominal trajectory may already have spent the state of the vehicle in a way that makes avoidance impossible. CMPC asks one extra question at every solve: if the contingency happens, can I still escape from where this command takes me?
For a linear time-varying system, let \(x_i^n\), \(u_i^n\) be the nominal prediction and \(x_i^c\), \(u_i^c\) the contingency prediction. The two branches can use different models—for example, normal versus low-friction dynamics:
Both start at the measured state. The key non-negotiable coupling is that the first command is the same, because the future event has not happened yet:
After the possible hazard is revealed, the controller may follow the contingency inputs. Before that moment, however, the nominal plan is judged partly by whether it leaves enough room for the emergency plan to work. A compact formulation is:
Here \(J_n\) rewards ordinary performance, \(J_c\) can simply reward a feasible escape rather than a comfortable one, and \(\lambda\) controls how strongly the nominal plan protects the contingency. In a safety-critical setting, the contingency constraints—not merely its cost—carry the guarantee.
In the canonical two-horizon CMPC formulation, only the current command is forced to match. Both branches start from the same measured state \(x_k\), and the command actually sent to the plant is shared:
But the next predicted states need not match. If the nominal and contingency models differ, then even the same command can produce \(x^n_1\neq x^c_1\). The future inputs \(u^n_1,u^n_2,\ldots\) and \(u^c_1,u^c_2,\ldots\) are allowed to diverge because the controller will re-solve after applying \(u_k\). In the special case of identical dynamics, the common input also gives a common one-step state; that is a consequence of the model, not an additional CMPC coupling constraint.
Choose \(\tau_b\) from the earliest time at which the real controller can know which world it is in, including sensing, state estimation, event classification, communication, and actuation delay. In the original CMPC setup, the next MPC solve occurs after the current command, so \(\tau_b=1\) is appropriate when the contingency can be detected before the next command is issued. If detection takes three sample periods, use a three-input common prefix. If the event time is uncertain, use several possible branch points in a scenario tree or impose a robust common prefix.
Yes—but the safest design is usually not to let the optimizer pick an arbitrary branch time. Let a monitor update the information state and trigger a branch when the evidence crosses a condition. The branch time then changes with the scenario, while the controller still respects causality.
Two useful versions are common. A measurement-triggered branch activates when a residual, fault classifier, belief probability, or sensor pattern indicates that the contingency is now likely enough to act. A viability-triggered branch activates when delaying the decision would make the contingency set infeasible—for example, when reachable safe corridors no longer overlap or the remaining braking/avoidance margin falls below a threshold.
This is a conceptual trigger, not a universal formula. The first term says “branch when the evidence is strong enough”; the second says “branch before waiting destroys the escape.” In implementation, one often computes a safe upper bound on postponement and branches no later than that bound. Recent Branch MPC work uses reachability to estimate the maximum feasible decision-postponement time and to prune or cluster scenario branches while preserving safety-relevant modes.
There is a subtle but important difference between dynamic branch activation and dynamic scenario generation. Activation chooses when an already-known contingency policy becomes actionable. Scenario generation changes which futures are represented as new measurements arrive; online scenario-tree adaptation and adaptive MPC address that problem by updating scenarios or uncertainty sets from current data.
At each MPC update, maintain two clocks. The first is an information clock: when could the real controller distinguish the scenarios? The second is a safety clock: how long can a common control prefix remain viable for all scenarios?
Here \(\tau_{\mathrm{info}}\) is the first time a causal detector or belief update justifies different actions. \(\tau_{\mathrm{safe}}\) is the latest time at which at least one common-prefix policy still keeps every relevant scenario inside its safe controllable set. If information arrives first, branch on evidence. If safety becomes tight first, keep the controls common but switch to a more conservative emergency-ready policy; do not pretend that scenario identity is known.
For interacting agents, a simpler heuristic is to delay branching while their predicted futures remain close, then branch once their reachable corridors separate beyond a task-dependent threshold. The threshold should be chosen from the control consequence—whether the same action remains safe—not merely from Euclidean distance. Recent reachability-based Branch MPC work uses corridor overlap to reduce scenario complexity and estimates the maximum feasible postponement time for branching decisions.
| Method | What changes online? | When it is useful |
|---|---|---|
| Event-triggered CMPC | Branch activation time | Fault evidence arrives asynchronously or detection quality varies |
| Reachability/viability trigger | Latest safe postponement time | Waiting is acceptable until geometry or constraints make escape impossible |
| Adaptive scenario-tree MPC | Scenario set and branch structure | New data changes which uncertainty modes are plausible |
| Belief-space or POMDP-style MPC | Branch probabilities and information state | The event is partially observed and sensing actions affect future knowledge |
| Approach | What it protects against | Typical trade-off |
|---|---|---|
| Robust MPC | All disturbances in a prescribed uncertainty set | Strong guarantees, often conservative |
| Scenario-tree MPC | Several possible uncertainty evolutions, with nonanticipative branch policies | More recourse, but tree size grows rapidly |
| Contingency MPC | A selected, identified emergency with an explicit escape trajectory | Less blanket robustness; more interpretable and targeted preparation |
CMPC is therefore selectively robust: it does not promise safety for every imaginable failure. It spends computation and performance margin on the failure that the designer has identified as consequential. The original CMPC formulation introduced an additional horizon in parallel with classical receding-horizon MPC and coupled the horizons through the first command; related multi-scenario MPC instead represents possible uncertainty evolutions in a scenario tree with nonanticipative inputs.
A document becomes a plan only when a tired person can use it at 3 a.m. Each recovery action should have an owner, a trigger, prerequisites, a safe stop condition, and a verification step. “Restore the database” is a goal. “On-call lead declares database-region loss; storage owner promotes replica B; verify checksum and write fence; announce read-only mode” is executable.
Separate three kinds of work. Incident response contains and investigates the event. Continuity keeps essential organizational functions operating. Disaster recovery restores technology and data. They interact, but one runbook should not pretend to replace the others.
| Plan element | Question it must answer | Concrete artifact |
|---|---|---|
| Trigger and authority | When do we stop normal procedures, and who can say so? | Activation matrix with named roles and alternates |
| Dependencies | What people, systems, data, facilities, and suppliers are required? | Function-to-dependency map |
| Strategy | How does the function continue if the primary path is gone? | Failover, manual, queue, cache, or alternate-supplier design |
| Communication | Who needs which fact, through which channel, and how often? | Internal/customer message templates and contact tree |
| Verification | How do we know the workaround is safe and complete? | Checks, acceptance tests, and exit criteria |
There is no universal “best” resilience control. Redundancy is useful when one component can fail independently; it is disappointing when both copies share the same region, identity system, power, vendor, or deployment pipeline. A manual fallback is powerful when the workload can be bounded and reconciled; it is dangerous when it creates untracked side effects.
| Failure mode | Possible strategy | Design question |
|---|---|---|
| Primary service unavailable | Failover, queue, cache, read-only mode | What can remain truthful while writes are delayed? |
| Data corruption or ransomware | Immutable/offline backups, point-in-time restore | Can recovery avoid reintroducing the bad state? |
| People or facility unavailable | Cross-training, remote work, alternate site | Can another person perform the function safely? |
| Supplier outage | Alternate supplier, inventory buffer, manual process | How quickly can the substitute be activated? |
| Loss of trust or information | Prewritten messages, status page, communication tree | What do stakeholders need to decide or tolerate next? |
Think in terms of independence. Two replicas in one blast radius are one dependency with extra steps. The right question is not “do we have two?” but “what common cause can still take both away?”
A plan contains assumptions: the backup is restorable, the alternate supplier answers, the credential works, the on-call person has authority, and the degraded mode is tolerable. An exercise is an experiment that tries to falsify those assumptions while failure is cheap.
Walkthrough: read the scenario and narrate actions. This finds missing owners, contacts, and decisions.
Functional test: exercise a subsystem or failover path with controlled scope. This finds integration and verification errors.
Operational exercise: involve the real people, communications, and constraints. This finds coordination and load problems.
Measure outcomes, not attendance: time to declare, time to preserve the essential function, data loss, unsafe actions avoided, and unresolved decisions. Feed every finding back into the BIA, architecture, or runbook. A plan that is never revised is a historical document.
The formula is intentionally informal. A service that eventually recovers but loses an irreversible promise may be less resilient than one that operates in a constrained, honest mode. Contingency planning is the design of that honest mode.