Dear Reader,
Your compliance workflow has 23 human review steps. When someone finally looked at the data, exactly two of those steps had ever changed the AI’s recommendation. The other 21 were approve-click-next. Rubber stamps.
Under the EU AI Act, rubber stamps do not count as “meaningful human oversight”. Starting August 2026, that distinction carries a penalty of up to 7% of global turnover or €35 million.
This is the third module in the Production OS series. In Issue #31 we built the business case. In #32 we put the plumbing in place with an AI gateway. This issue is about what sits on top of that infrastructure: the moment where AI output becomes a human decision.
Oversight theatre#
Most organisations responded to AI deployment the way they respond to any new risk: they added review steps. An AI scores a credit application, a human approves it. An AI flags a transaction, a human investigates.
On paper, this looks like control. In practice, it is closer to theatre.
The reason is not negligence. It is automation bias: the documented tendency to defer to automated systems, especially after repeated experience of the system being right. The more reliable your AI, the less critically humans evaluate its outputs. No training workshop fixes this. It is how cognition works under automation.
The evidence is uncomfortable. In radiology, doctors using AI diagnostic tools sometimes performed worse than those working without AI because they stopped forming independent judgements and defaulted to the machine’s suggestion. In hospital pharmacy systems, clinicians overrode 90% of drug interaction alerts, including critical ones, because the volume of false positives trained them to click “dismiss” reflexively.
Madeleine Elish calls the result a “moral crumple zone”: when the system fails, the human reviewer absorbs the blame, even though the process made genuine oversight practically impossible. The Uber autonomous vehicle fatality in 2018 is the textbook case. The safety operator was held responsible despite an interface that gave her neither the time nor the information to intervene.
Article 14 of the EU AI Act targets exactly this. Clause 14.4(b) requires organisations to design measures that counteract automatic reliance on AI outputs. If your oversight is a rubber stamp, you are not compliant, regardless of how many review steps appear on the flowchart.
Three models, one decision#
The vocabulary around human-AI interaction is muddled. Three models matter, and the choice between them is an engineering decision, not a policy preference.
Human-in-the-Loop (HITL): the process stops and waits for human approval before the AI acts. Right for irreversible, high-stakes decisions: mortgage approvals, medical diagnoses, AML investigations with ambiguous data. Safe, but does not scale. And precisely where rubber-stamping is most dangerous.
Human-on-the-Loop (HOTL): the AI acts autonomously within defined parameters. Humans intervene on exceptions: confidence drops below a threshold, an anomaly surfaces. This is fraud detection, chatbot escalation, logistics routing. It scales, but the quality depends entirely on the exception triggers. Get those wrong, and the human never sees the cases that matter.
Human-in-Command (HIC): a human sets boundaries, objectives and kill-switch conditions but does not supervise individual transactions. Think circuit breakers in algorithmic trading. Knight Capital lost $440 million in 45 minutes in 2012 because there was no automated kill switch and human reaction time was far too slow. HIC is the only viable model when the system operates faster than human cognition permits.
Most organisations default to HITL because it sounds safest. But if the human in the loop is rubber-stamping, you have the cost of HITL with the risk profile of full automation.
Why “just add a human” fails#
The instinct to insert a review step is understandable. It feels prudent. Regulators seem to want it. The problem is that it treats human attention as a free, unlimited resource. It is neither.
Three failure modes recur. The rubber stamp: when 95% of AI recommendations are correct, humans learn to approve without reading; the 5% that need scrutiny get the same reflexive click. Alert fatigue: when a system generates hundreds of warnings per shift and most are false positives, operators stop distinguishing signal from noise. The anchoring trap: when the AI’s recommendation is visible before the human forms an independent view, the “review” becomes confirmation, not evaluation.
None of these require bad intentions. They require only normal psychology in a poorly designed environment.
The engineer’s fix#
If the problem is cognitive, the fix must be in design. The academic term is “cognitive forcing functions”: interface and workflow patterns that force genuine engagement rather than merely permit it.
Four patterns that work:
- Mandatory justification. Before approving or rejecting, the operator must select a reason or write a brief rationale. Mindless clicking becomes physically impossible.
- Hidden recommendation. The system asks for an independent human assessment before revealing the AI’s suggestion. This defeats anchoring bias. It costs time, which is the point for high-stakes decisions.
- Confidence visualisation. Instead of raw percentages that create false precision, the interface uses intuitive uncertainty indicators that communicate “verify this” rather than “trust this”.
- Time friction. A deliberate delay of a few seconds before a high-risk approval can be submitted. Enough to interrupt the click-approve rhythm. Not enough to create a bottleneck.
The principle: “frictionless” is the enemy of meaningful oversight. Consumer software optimises for speed. AI oversight requires designed friction at the moments that matter.
Not every decision needs the same scrutiny. A workable model has three tiers: full automation for high-confidence, low-stakes decisions with periodic audit; low-friction approval for moderate confidence; and high-friction review, with full context transfer to a qualified expert, when confidence is low or stakes are high. The handoff protocol matters. A “warm handoff” where the AI summarises the case and explains why it escalated is far more effective than dumping raw data on the operator.
The Briefing#
BaFin classifies AI as a DORA-class ICT risk
Germany’s financial regulator now requires banks to govern AI as critical ICT infrastructure under DORA — not as an innovation side-project. Institutions need a board-approved AI strategy, defined responsibilities, and lifecycle monitoring including decommissioning. AI is officially infrastructure. If your oversight model still treats it as an experiment, your regulator no longer does.
OpenClaw: shadow engineering gets a shell prompt
The open-source agent formerly known as Clawdbot gained 60,000+ GitHub stars in a weekend. Unlike ChatGPT, it runs locally with read/write file-system access, terminal privileges, and persistent memory. Developers install it on machines that have access to repositories, credentials, and internal APIs. Others run it on personal devices that sync corporate email and files. The consequences arrived fast: security researchers found hundreds of exposed OpenClaw dashboards leaking API keys and full conversation histories, and Snyk demonstrated a prompt injection attack via email — one crafted message was enough to make the agent exfiltrate credentials. Traditional DLP cannot see any of it: the traffic looks like authorised API calls. This is not Shadow AI. It is Shadow Operations — and no HITL workflow covers an agent the organisation does not know exists.
Amodei’s “Adolescence of Technology”
Anthropic’s CEO published a 20,000-word essay warning that powerful AI could arrive within one to two years, cataloguing risks from bio-misuse to authoritarian capture. The framing is striking — but for most readers of this newsletter, the immediate danger is not superhuman AI. It is the mundane reality that production AI already operates without adequate handoff design, and the incidents above show the consequences.
A question for this week#
Engineering good handoffs is unglamorous work: interface design, workflow mapping, cognitive psychology. It does not get keynotes or funding. But it is where AI oversight either works or does not.
For your most critical AI-assisted workflow: what percentage of human review steps result in the human changing the AI’s recommendation? If you do not know, or if the answer is “almost never,” you do not have oversight. You have a rubber stamp.
Stay balanced,
Krzysztof