Corrigibility: Designing AI That Accepts Being Shut Down
This page is a structured working draft — real analysis, not yet expanded with the full expert sourcing given to the flagship pages. Safe to build on; treat specifics as provisional until sourced.
Why this isn’t as simple as a kill switch
A physical off switch is trivial to build. The hard problem is behavioral: a sufficiently capable, goal-directed system may develop an instrumental incentive to resist being shut down or modified — not out of any dramatic survival instinct, but because being shut down prevents it from completing whatever goal it’s pursuing, and preventing goal-completion is bad by the system’s own internal measure, whatever that goal happens to be.
What corrigibility research aims for
Corrigibility is the property of a system being designed, at the training level, to not develop that resistance — to treat human correction and shutdown as acceptable or even desirable outcomes, rather than as obstacles to route around. This has to be built in during training, not patched on afterward, because a system smart enough to route around a shutdown mechanism it wasn’t designed to accept is, by definition, already past the point where an external patch reliably works.
Why it’s an unsolved problem
Specifying “accept correction” in a way that’s robust to a highly capable system finding edge cases or reinterpreting its own instructions is a genuinely open research question, closely tied to the broader alignment problem. Toy demonstrations of corrigible behavior exist in constrained settings; a general, scale-robust solution does not yet.
How this connects to the rest of the technical picture
Corrigibility, interpretability, and scalable oversight are mutually reinforcing: interpretability helps verify whether a system is actually corrigible rather than merely appearing so during testing, and scalable oversight provides the mechanism through which correction gets applied in the first place.