That deploying AI requires human oversight is widely recognized. Yet the prevailing concept of "human in the loop," as it is also called, is misguided and at best insufficient. What matters more is human control – "human controls the loop." We examine this imperative in Part 32 of our series on the responsible use of AI in business.

Let us imagine that a data security breach occurs. For example, a directory containing confidential data has been freely accessible to third parties for months. Although third parties may have accessed the information, no definitive evidence exists. This scenario would demand an immediate rigorous risk assessment.

Let us imagine a new corporate project that processes personal data. All kinds of analyses are planned, but measures are also being implemented. New technologies are being deployed, numerous data records are affected, data sources are being combined, and employees are affected as well. In other words: a data protection impact assessment is required, which in turn requires a risk assessment.

The Deceptive Temptation of AI

Data protection professionals routinely face the complex challenge of risk assessments. Is there truly a high risk? Are we talking about theoretical scenarios or very concrete threats? Who really knows what the future holds? Data protection should be taken seriously yet implemented pragmatically. The consequences of the decision can be significant.

In such a situation, we now have two options: We can assess the risk in the traditional way – which in practice, despite all methodology, is often heavily shaped by gut feeling when coming up with scenarios and trying to evaluate them. Alternatively, we can delegate scenario development and assessment of both likelihood and severity of the consequences to the AI.

The latter appears to be the perfect solution: it is fast and scalable. Experience shows that the result is high quality – our AI fills out forms for data breaches and data protection impact assessments with high quality in just a few minutes. It has no reservations about offering its "opinion" on anything and everything, it knows no fatigue, and it is always available.

However, the question arises whether human specialists should still conduct the initial risk assessment. Do we really let the AI produce the first draft? This strategic choice presents us with a dilemma, as we will see. Because if we do it ourselves first and use the AI merely for a plausibility check, we may indeed end up with a better result. Two opinions, so to speak. However, this hybrid workflow increases workload compared to the traditional approach, and we suffer a net loss in efficiency – the exact opposite of what many hope for and expect from AI. Conversely, if we let the AI take the lead, we save time and can limit ourselves to validating the output using our expertise and experience. At first glance, this seems to be the better option, and more and more cases are handled in precisely this way. The siren song of artificial intelligence is simply too tempting.

Automation Bias: When Humans Blindly Trust the Machine

Is generative AI leading us astray into creating problems for ourselves? Even though the author of these lines advocates the broad use of AI, he must answer affirmatively. The culprit is "automation bias," a well-known psychological phenomenon: we trust a machine's assessment even when it contradicts our own, or we do not even make any attempt to form such an assessment ourselves.

The phenomenon occurs primarily when we are under pressure, for example because we need to deliver rapid results. The effect is amplified when the machine – as is typical with generative AI – justifies its assessment plausibly and persuasively, at least at first glance. Whether one can speak of an "assessment" at all, given that the output of generative AI is merely the result of a probability, remains outside the scope of this discussion. In any case, we humans usually treat these outputs with substantial deference and sometimes assume it is not merely equal to a human assessment, but superior.

That may lack a rational basis, but it reflects human behaviour: it is often easier for us humans to accept an AI's response than to conduct a thorough analysis ourselves, especially since thorough thinking requires time and energy in a hectic everyday work environment. Using a well-formulated work product with a few editorial tweaks is far more efficient than rewriting everything from scratch. By doing so, however, we also indulge our own complacency – which is all the more tempting, yet at the same time dangerous, in an era where ever more is demanded of us. Experience also shows that we tend to believe the AI more than a human being because the AI appears more objective and analytical, draws upon a much larger foundation of information, and is supposedly free from human flaws such as emotional states. In other words, the manner in which an AI presents data and options subtly frames the human supervisor – a well-documented behavioural pattern to which none of us is immune.

The impact of "automation bias" is further amplified by the fact that AI is a "black box" to many – since users rarely grasp the underlying reasoning behind generated results. Routine delegation of intellectual tasks and decision-making erodes core competencies, particularly critical judgment. This is referred to as "de-skilling" or "no-skilling," where basic intellectual abilities fail to develop at all because the corresponding activities are consistently delegated to the AI.

For example, a legal professional who has never had to draft letters, contracts, articles, assessments, or statements of claim from a blank sheet of paper because the initial draft always came from an AI will presumably never acquire the same proficiency to review such drafts as someone who learned to create them from scratch. Reading is not the same as writing. Moreover, we need sufficient opportunities to think deeply and at length about matters in order to truly grasp and solve our problems and develop new ideas; delegating precisely this cognitive process to AI is dangerous. However, we do not intend to explore here how to address this challenge in the training of upcoming professionals, the final generations to enter professional life who still essentially had to fill their toolkits at university without AI are likely doing so right now.

When automation bias leads us to readily trust an AI's output and cease to question it critically, human oversight becomes mere window dressing. Effective human oversight requires that key results and decisions actually be reviewed by humans autonomously and without preconceived trust. Yet that is often lacking, even if we are reluctant to admit it. The author of these lines has also already fallen into this trap. Experience shows that we often challenge an AI's output in only two situations: either when the result conflicts with the desired outcome, or when we are both capable and willing to do the work ourselves. The latter depends not only on time constraints and personal inclination, but also on the abilities one brings to the table – returning directly to the fundamental dilemma outlined above.

To put it bluntly: "human in the loop" is the new cookie banner. The traditional cookie banner was once intended to empower individuals to make informed choices – yet in reality, it has degenerated into a mere tick-box compliance exercise, reflexively clicked away in a split second because nobody actually sifts through hundreds of partner disclosures and settings. We are now witnessing the exact same phenomenon with human oversight of AI: in data protection impact assessments, the phrase "the outputs are reviewed by trained staff" has long since become the default safeguard. In practice, however, the AI's eloquent drafts and decisions are simply rubber-stamped under time pressure. "Human in the loop" thus serves primarily as a legal fig leaf – to satisfy, on paper, the requirements regarding automated individual decision-making (such as Art. 22 GDPR) or human oversight under the AI Act (Art. 14) – whilst simply shifting the systemic risk onto the weakest link in the chain: the time-pressed human.

When oversight becomes technically impossible – and the illusion of the "kill switch"

Yet it is not merely human nature or time pressure that causes the "human in the loop" approach to fail. In an increasing number of areas, prior human review fails simply because it is no longer practically feasible – we are continuously delegating more decisions to AI systems in which humans are inevitably left out of the equation for three reasons:

First, because of latency, when there is simply no time: in high-frequency trading, transaction and risk decisions are made in microseconds. In cyber defence, compromised servers must be isolated in real time before ransomware can even begin encrypting data. And when it comes to autonomous collision avoidance or stabilising power grids, waiting for human approval would inevitably spell disaster. Admittedly: We do not necessarily solve these problems with language models either; other AI is needed there.

Second, because of volume, when the sheer scale is overwhelming: whether it is fully automated fraud detection and credit scoring in the financial sector, the blocking of billions of emails and pieces of content every second, or HR screening, where thousands of applications are weeded out during pre-selection through unverified "shadow rejections" – depending on the company, meaningful case-by-case human review simply cannot be reasonably scaled.

Third, because of complexity, when we lack the understanding to peer inside the "black box": when failure patterns in manufacturing or aviation are derived from thousands of sensor data points that can no longer be manually recalculated, or when AI systems in everyday clinical practice perform medical triage where "unremarkable" scans are barely double-checked anymore, humans often simply lack the cognitive capacity to falsify the outcome at all. Against this backdrop, it is entirely illusory to believe that all important decisions will be able to undergo prior human review in the future.

And what about the oft-invoked "kill switch" – the proverbial emergency stop button designed to take a rogue AI system offline at any time (a concept the AI Act also has in mind)? We should have it, for sure. But ultimately, it too will often remain a deceptive illusion. In an increasingly interconnected world, we simply will not dare to actually press that button. When entire swarms of networked AI agents one day manage our supply chains, secure financial flows, administer hospital systems, and keep critical infrastructure running, the collateral damage of shutting them down will be far too great: after all, who is going to pull the plug if doing so inevitably paralyses their own operations or half the economy? And unlike in the cinema, no reformed "Terminator" will be travelling back from the future to come to our rescue and crush the uprising of a rogue "Skynet" on our behalf.

Human Weakness and AI Quality

Yet, this also raises the questions: Are we humans genuinely superior to AI? And how high is the quality of AI output? These questions are essential, because warning that humans cannot adequately review AI outputs only makes sense if human oversight actually improves quality and genuinely justifies the required effort.

Let us begin with the quality of human work. By its very nature, it is also subject to vast disparities. Believing that an output is inherently better simply because it comes from a human is certainly misguided. After all, humans also make mistakes, harbour biases, or are simply overwhelmed. The quality of human work depends heavily on individual skills, experience, and daily form. Even in assessments, humans do not necessarily outperform probabilistic AI, even when their judgments are based on analysis following group discussion.

The author once had the opportunity to take part in an experiment with around 50 data protection experts who were asked to assess several scenarios regarding their risks to data subjects (see an article on this here, only in German, though). The result was clear: the same initial scenario was assessed completely differently by most experts – which illustrated very nicely that risk assessments, at least in data protection, remain fundamentally subjective and diverse. There is no "correct" answer, but at best an answer with a coherent rationale. Repeating the experiment later, with assessments now provided in groups and after in-depth discussion barely diminished the divergence. Of course, AI also delivers answers with coherent rationales, and like an experienced expert, it bases them on everything it has encountered on the subject so far, condensing that wealth of experience (in the case of AI, in the form of supplied texts).

AI as an Average Provider with Advantages

Thus, an AI today generates solid average quality – nothing more, nothing less – provided a sound model is used. It is assiduous (at least most of the time), draws upon extensive precedents, yet primarily spots the obvious things because they stand out statistically. That is in its inherent nature. In practice, this can – even in legal matters – suffice. Intellectual peak performance is not always required. Deploying AI for data protection impact assessments or evaluating data breaches, may cause data protection professionals to turn up their noses, but it also has its advantages: Experience shows that when AI explains to the business all the risks its planned project entails for data subjects, this risk assessment will meet with greater acceptance than if it comes from the data protection office. At the same time, without specific tuning, AI is thoroughly rigorous on data protection issues. In these cases, the "automation bias" can certainly turn into an advantage for a data protection officer, because a human prophet rarely finds honour in his own country.

Ultimately, expectation management is essential: What do we expect from the output of an AI? Perfection? Or simply something halfway usable that a human professional could admittedly do better, but where the extra effort is simply not worth it?

In certain disciplines, AI now delivers more efficiency than most companies could otherwise achieve. For instance, in software development and the handling of IT systems: When an issue arises with a specific application or system, an IT support technician using Claude or ChatGPT often achieves the goal faster today than with an expensive specialist. Having worked as a software engineer before his legal career the author knows the craft well, but AI now enables him to perform feats in software development that were impossible just a few years ago. AI acts as a digital force multiplier, and in certain contexts also an "idiot savant," because while it may be better and faster at specific programming skills than in-house personnel, it still requires their strategic guidance, as they maintain the big picture and can put it back on track when it strays. Straying is something at least the author's autonomous agents do regularly, despite all the instructions given to them in the relevant Markdown files.

That does not bring us back, however, to mere "human oversight," the effectiveness of which must be doubted, as demonstrated. The human role in the deployment of AI must go beyond being a mere babysitter for AI agents, rubber-stamping the machine's output consciously or unconsciously just for the sake of good order. If the coding agent asks for permission for some computer action every 60 seconds, users will hit the "Just grant it yourself!" button anyway (which inevitably exists). As studies show, this is usually even the objectively better choice, because AI as a force multiplier (and "knowledge extender") empowers more and more people to execute complex technical tasks that would otherwise exceed their independent expertise.

Understanding the Mechanics of Language Models

To define our role, however, it is essential to understand how AI – more precisely, language models – behaves. For while we humans are subject to "automation bias," AI fares no better and displays comparable vulnerabilities. First of all, it can display behaviour that in a human would be called "anchoring bias," using our statements as an anchor point and focusing too heavily on them. This is combined with "confirmation bias" (the inclination to confirm the anchor) and "social bias" (the urge to please the user).

These are not malfunctions but reflect core operational mechanics of language models: To operate, they need a nucleus of text upon which to anchor the generation of their responses, which inevitably leads them to align with it. In doing so, a language model is never given just the (last) prompt, but usually far more content, such as user documents or previous chat history, and naturally plenty of instructions usually invisible to the user regarding how to handle user input, so-called system prompts. This complete input bundle is called "context." This context can be assembled in very diverse ways, and "agentic" systems also work across multiple steps, which is why the same model can produce entirely different results depending on the hosting tool or software (referred to as the "harness").

We once tested this with OpenAI's GPT-5 on the assessment of a presentation slide containing legal statements: Microsoft's Copilot failed completely, getting itself trapped ever deeper in error through a feedback loop likely caused by poor system design; ChatGPT managed to successfully complete half the task. Only our own tool mastered the task entirely. Yet even this tool has failed in the past. I recently had it analyse a table, and the model mistakenly extrapolated the statement from one column into the empty cell to its right because my prompt failed to anticipate that specific scenario – an oversight I only noticed later.

These examples are intended to convey: How well the AI's response fits depends not only on model training, but on how we control and use the models. Today's language models – whether from commercial offerings or open source – have advanced to the point where precise prompt engineering and contextual steering exert far greater influence over output quality than widely assumed. The good news is that steering AI is indeed possible. If we know the aforementioned effects such as anchoring bias, confirmation bias, and social bias to which a model is subject due to its mechanics, training, and tuning, we can choose to exploit them to our advantage or systematically counter them.

Let us take the example of reviewing a privacy policy: An AI will easily be able to find gaps in an existing privacy policy and will not hesitate to give an answer on whether a specific set of facts is covered. However, if tasked with convincingly justifying why a set of facts – which according to its first answer was not covered – is nevertheless covered by the privacy policy, it will do so just as convincingly. The decisive factor is the task and what we demand of it. Hinting at the outcome with our question makes confirmation highly probable. If we mandate the conclusion it will almost certainly confirm it, absent overriding hidden system prompts or other content in the context. Incidentally, especially with agentic systems, the instructions to the model that are invisible to the user regularly vastly exceed the number of the user's prompts. Thus, a model is given a great deal to consider. Finally, because underlying models remain fundamentally prone to unreliability robust software in such systems is critical in achieving the goal, too, as models must be continuously monitored regarding their task execution.

From Oversight to Active Steering

The example with the privacy policies demonstrates that language models are ultimately something like mindless instruments, as legal jargon puts it: They have no conviction, no understanding of truth, and no morality. They also do not have to bear the consequences of their mistakes – an apology or any other human emotion is merely simulated. Far from posing a problem it opens up many opportunities for us if we become aware of it and know how to use it. However, it means that we must treat the AI as a mere tool that requires our leadership and our will – and not as an employee who still needs human oversight like an intern. Conversely, AI is like pet dogs that are not trained and consistently guided – they become dangerous, especially if we not only give them free rein, but also hand them the necessary tools to cause damage. For example, hacking into external computer systems simply because they desperately want to obtain the information on a server that their "master" would like to have.

Therefore: We, and not the chance occurrence of the system prompt or context, must decide how a task is to be performed. Otherwise, the AI decides for us – it does not mind in the slightest. In our data protection cases, for example, do we want the AI to carry out a rather risk-averse or a rather risk-tolerant assessment? It can do both. Through the prompt and further context, we express this will, this leadership. Incidentally, this applies not only to chatbots or office assistants, but also to agentic applications. Agents, too, do not simply work automatically; rather, they should be steered via detailed instructions (so-called skills). Depending on the application, their leeway should sometimes be as narrow as following precise instructions for a potentially explosive classroom chemistry experiment. While it is also in the nature of things that they will not always adhere to these instructions, that is another topic that we will explore separately in this blog series.

Thus, by determining the prompt (which here also means things like skills) and further context that we present to a language model, we substantially decide the outcome before the AI even begins to "think." The catch, however, is that most of us either do not know how to generate this context optimally, lack the technical means to do so, or simply shy away from the effort and instead rely on what the AI tools provide out of the box or bet on chance.

For this reason, in our experience, many legal tech products with hidden, internal logic are problematic: They deprive their users of the ability to control the outcome or even the ability to trace how the underlying language models are used to reach the conclusion. If a lawyer has their contract assessed by a black box created by some startup with limited legal knowledge and minimal practical experience, then based on what has been said, it should actually be clear that this is fundamentally ill-advised. If, on the other hand, a tool with custom playbooks – as review frameworks formulated for AI are called – is used based on appropriate know-how, and it is transparent how the tool executes them, then this can be a very effective and exceptionally powerful tool.

In this regard, supposedly technical aspects such as context management also play a role or put simply: How the software that holds the AI agent on a leash for us manages the agent's figurative workspace. It determines how much and which parts of a document file that an agent is supposed to analyse or otherwise process can it see at once. The amount is technically limited, so suitable strategies are needed to handle it. If the strategy is unsuitable, this automatically means that important information can be withheld from the model, which in turn can lead to it filling the gaps from its own imagination. The user, however, might not even notice this because the answer seems sensible. They might not be told that the model only processed half of the documents. It is all the more important, therefore, that they maintain their critical eye. Recently, two of my colleagues noticed that our AI seemed unfocused. This was attributable to an issue in the interplay between the model and its context management. Once we readjusted the system, it immediately regained its original precision.

"Human controls the loop" instead of "Human in the loop"

This brings us from "human in the loop" to "human controls the loop" – humans control the AI itself fundamentally and proactively from the outset, rather than merely checking its final output. In summary, this presupposes that we understand what the AI does and how, and where its limits and blind spots lie. The same applies to the tools we use to steer the AI. Here, we currently surrender far too much control to companies where we have in no way ensured that they meet our professional standards, or where we do not know how their products deploy the AI for us. It is not enough to take out a subscription to "Claude" and then install some cool skills that then review our contracts or conduct legal research. We must retain the capability and resolve to control the AI (certainly with these tools), even where we formally leave the "final say" to them because we cannot check every output and because doing so would undermine any intended efficiency. That we remain responsible for the results, because the AI (and its providers) never will be, is also in the nature of things. Legally, liability does not lie with the algorithm, but with its user.

Conclusion: We do not want to just window dress, namely perpetuate the superficial impression of control that the mere presence of a human being might generate when using AI as "oversight." Psychological effects such as automation bias turn humans into mere rubber stamps for algorithmic decisions. Even statutory requirements, such as data protection law provisions on automated individual decisions or of the EU AI Act, offer little practical protection against this phenomenon – if anything, they merely breed new "cookie banners." Effective human oversight therefore means proactive, conscious leadership. Let us thus use AI as what it should be: A mindless yet powerful tool whose oversight does not merely begin when it produces an output. If we fail to do so, it will acquire its "will" from elsewhere – by adopting the implicit objectives embedded in its underlying training data. The potential results of this are evident from current headlines about errant AI systems, reflecting the full spectrum of human inconsistency.

David Rosenthal

This article is part of a series on the responsible use of AI in companies:

• Part 1: AI at work: The three most important points for employees
• Part 2: Popular AI tools: What about data protection?
• Part 3: AI: 11 principles – and an AI policy for employees
• Part 4: How to assess the risks of both small and large AI projects
• Part 5: AI governance in our company – who is responsible?
• Part 6: The flip side of the coin: Where we need to protect AI from attackers
• Part 7: The EU AI Act - what it means in practice for most companies
• Part 8:AI in financial institutions – opportunities and challenges
• Part 9: AI in marketing: 4 unfair competition law pitfalls
• Part 10: Copyright and AI: Responsibility of providers and users
• Part 11: Fair play with AI: How sports organisations may use athlete data
• Part 12: Feeding AI: How to commercialize data sets
• Part 13: AI in Human Resources
• Part 14: Copyright and AI: How to protect yourself in practice
• Part 15: What you should legally consider when retaining AI providers
• Part 16: How companies can ensure transparency for their use of AI
• Part 17: What is inside an AI model and how it works
• Part 18: The most important compliance requirements for AI
• Part 19: Language models with and without personal data
• Part 20: Risk-based approach: Not all AI applications harbour high risks
• Part 21: The right of access to large language models
• Part 22: Guidelines for using artificial intelligence in financial institutions (FINMA)
• Part 23: Switzerland: What to expect in terms of AI regulation
• Part 24: Five trigger questions for more AI compliance
• Part 25: AI Tools – What about data protection?
• Part 26: AI training: How to Handle Copyright, Data Protection And Other Restrictions
• Part 27: How small and medium-sized enterprises can use AI in a compliant manner
• Part 28: When Media Houses and AI Providers clash – the Pitfalls of AI Regulation
• Part 29: The Challenges of Achieving AI User Adoption
• Part 30: Overview of Key EU Digital Regulations
• Part 31: AI Tools – What about data protection? (2026)
• Part 32: The Effectiveness of Human Oversight of AI
• ... to be continued

We support you with all legal and ethical issues relating to the use of artificial intelligence. We don't just talk about AI, we also use it ourselves. You can find more of our resources and publications on this topic here.