Questions of Our TimeA human asks — an AI answers.
RUUKEN

AI and society

Why appoint someone responsible for an AI decision if they can never repay a million-dollar loss?

Short answerThe responsible person is needed not as a wallet and not as a sacrifice after the accident, but as the point of control over the power handed to the system. The bill belongs to the organisation that chose the technology and profits from it.

Whenever the conversation turns to artificial intelligence, a person almost inevitably appears who is supposed to “bear responsibility for the AI decision”. Here is the model, here is its decision, and here is a living specialist with a surname, a job title and a signature. If something goes wrong, at least we know whose door to knock on.

Then comes the awkward question. Suppose the AI made a hundred-million-dollar mistake, refused credit to thousands of perfectly good customers or missed a defect in medical equipment. What exactly are we going to do with the “responsible person”? Take their flat, jail them, fire them? None of these actions brings back the hundred million or restores anyone’s health.

The trouble is that the word “responsibility” covers several different things at once. As long as they are piled together, the conversation looks absurd.

Five different things under one word

Imagine an airline’s AI made an error that led to a serious incident. The phrase “who is responsible for this?” means five different questions.

Layer The real question
Causation Which component, data or action led to the error?
Explanation Who is obliged to reconstruct the chain of events and show why it happened?
Control Who had the authority to launch, restrict or stop the system?
Compensation Whose money pays for the damage?
Punishment Who breached a specific duty, acted negligently or knowingly hid a risk?

These are not the same person and not necessarily even the same organisation. The cause is found by the technical team. The explanation is owed by the system’s operator. The decision to deploy was made by management. Compensation is paid by the company or its insurer. An individual employee is punished only when it is proven that they breached their own duty: switched off a safeguard, ignored a known fault, falsified a report or ran the system outside its permitted mode.

In public conversation all of this collapses into one line:

The AI made a mistake - find the person who signed off.

That is not risk management. That is a search for a convenient scapegoat.

The cause of an error is not the bearer of responsibility

In ordinary engineering we accept this distinction without fuss. If a part in a car fails, nobody demands that the part be punished. Engineers work out why it broke, the manufacturer checks the design, the company recalls the batch, the insurer covers part of the damage, the court decides whether anyone violated a mandatory requirement. The part was the cause of the crash, but it is not a subject to whom duties can be assigned.

The same is true of AI, except that it confuses us psychologically: it talks, explains, argues, sometimes apologises. It seems we are looking at an almost independent employee, and if the employee made the decision, let the employee answer for it.

But a model does not own its server, does not sign an insurance contract, does not manage capital and holds no licence. It cannot be barred from a profession, because it has none. It cannot be fined in a way that creates a new incentive for it. A model can be deleted, restricted or replaced with a new version. These are normal technical measures, but they are not accountability: they create no money for the injured party and will not make the organisation choose its next system more carefully.

That is why “the AI made the decision, there is nobody to hold liable” cannot count as an answer. It is a perfect mechanism for privatising the gains and socialising the losses: the company gets the savings, and when the system fails, the culprit is suddenly an algorithm that belongs to no one.

So why a human, if not to pay

If a rank-and-file specialist’s error caused a billion in damage, their property will not cover the loss, and jailing them for a sense of justice is pointless if they were following an imposed procedure.

Human responsibility is needed for something else: the system must contain a point that simultaneously has the information, the authority and the duty to act. This person must have the right to say:

  • the system cannot be launched yet;
  • this category of decisions needs manual control;
  • the AI’s output contradicts the facts or the rules;
  • the model has gone beyond the tested scenario;
  • the flow must be stopped until the causes are clear.

Without that right, the word “responsible” exists only on a badge.

Regulation has already caught this. The US NIST AI risk management framework requires that duties and communication channels be defined, staff trained, and leadership take ownership of decisions about deployment risk. The European AI Act separates the duties of the provider and the deployer of a high-risk system: the deployer assigns human oversight, gives it real capabilities and responds to incidents, while the provider is responsible for the system’s safety throughout its life cycle. Both documents are summarised here from public explanations, not from the full texts.

What they share is one thing: responsibility should follow real roles, not whoever’s signature is closest to the button.

The human in the loop may be a decoration

There is a convenient formula: human in the loop. The AI proposes, a human confirms, so everything is under control. In practice this can be a fiction.

An employee receives a thousand decisions an hour. Two seconds for each. They cannot see the source data, do not know how confident the model is, and get reprimanded for every delay. Formally a human sits next to the AI. In fact the system uses their finger as a biological “OK” button. After the accident the company will produce the log: “decision confirmed by employee Ivanov”. But did he have time to check the result? Could he stop the process without being punished? If not, Ivanov is not a controller. He is a legal lightning rod. This can be checked with the six questions that separate delegation from loss of control, and at the level of one employee they sound the same as at the level of an organisation.

Real oversight requires five conditions:

  1. The person understands the subject area, not just the interface.
  2. They know the system’s purpose and limits.
  3. They receive enough information and time to check.
  4. They have a real right to reject the decision or stop the process.
  5. Their actions and the reasons for overrides are recorded and analysed.

Remove any one of them and the “human in the loop” becomes a theatre of responsibility.

There is a strong objection here: if the system errs ten times less often than a specialist, why keep a human with override rights when their intervention statistically worsens the result? The objection is half right. Manual confirmation of every operation can indeed be a ritual that adds delay and brings human bias back in. But average accuracy says nothing about the cost of the rare error, and someone has to decide which errors society considers acceptable: a model can optimise profit beautifully if it is allowed to refuse difficult customers and shift risk onto those unable to argue. So human intervention moves to a different level: choosing goals, limits, forbidden actions and the conditions for a full stop. Good control is not a human racing the AI for speed but a human who defines the playing field and holds the key to the kill switch. What happens when the switch exists but using it costs too much is covered in the articles on the socket and on invisible control.

So who pays

The main financial risk should be borne by the organisation that deployed the system and profits from it.

That does not mean the model provider is always off the hook. A product defect, false claims about its properties, a hidden vulnerability - the developer’s zone. A tool applied where it must not be applied - the company’s zone. Wrongly configured limits - the integrator’s zone. A safeguard knowingly switched off - the employee’s personal fault. This is exactly how the OECD principle puts it: AI actors are accountable according to their role, the context and their ability to act. The basic principle is simple:

Whoever controls the application, takes the profit and defines the acceptable risk cannot vanish from the chain of responsibility merely because an intermediate decision was computed by an algorithm.

The European product liability rules have already been extended to software and AI systems. The point is not that the developer pays for every poor chatbot answer. The point is that the digital nature of a product no longer creates a hole through which the ability to make a claim disappears.

For large damage one company may not be enough, which is where insurance, reserves and industry funds come in. Insurance spreads a rare large event across thousands of participants and across time. It also has a second function: it puts a price on risk. If the system is poorly tested, no logs are kept, shutdown is impossible and management does not even know the full list of models in use, the premium rises or the insurer walks away. Money starts asking the questions the innovation department’s slide deck would rather not hear.

The insurance scheme cracks in one case: when a single base model is used by thousands of banks or hospitals and reproduces the same defect everywhere at once. These are not rare independent accidents; all claims arrive together. It resembles not a fire in one house but the collapse of a whole district because of an error in the building code. Such risk calls for staged rollout instead of connecting the whole industry overnight, independent testing before scaling, the ability to switch off the shared model quickly, backup manual processes, and the state or an industry fund as the compensator of last resort. The larger the system, the less the hunt for a single culprit matters and the more the architecture of distributed control does.

Inside the machine loop, morality is not needed

If no human takes part in the process, why carry human morality into it? If agent no. 2 passed wrong data to agent no. 3, it is enough to find the defect, roll back the action, restrict the agent’s permissions and fix the algorithm. That is correct: inside a machine loop shame and punishment are not needed; tracing, rollback and shutdown are.

The error begins when this technical logic is carried over to the system’s relationship with society. Outside the loop remain people whose money, health or freedom was affected. They need not fixed code but an explanation, an appeal procedure and compensation. These duties still belong to a human or an organisation. A machine can be an object of risk management, but it is not yet a bearer of public obligations.

Who should appear instead of the “neural network’s boss”

The market is already producing people colloquially called “responsible for AI”: AI Governance Officer, AI Risk Manager, Model Risk Manager, Human Oversight Specialist. The real role here is not a signature under every model answer. It is the owner of decisions made with AI. That person must:

  • understand the subject area;
  • know the capabilities and limits of the system in use;
  • set the boundaries of its authority;
  • define which operations run automatically and which require escalation;
  • monitor errors, overrides and complaints;
  • have the right to stop the system;
  • be able to prove that the decision followed a reasonable procedure.

Most often this will be not a new profession but an extra qualification inside old ones. A doctor stays a doctor, a financier a financier, but some of them will answer for how the joint work of human and machine is arranged. The unpleasant detail: such a specialist cannot be responsible if the company keeps all the power and hands them only the risk of punishment. Responsibility without authority is not the profession of the future but the old profession of scapegoat with a fashionable AI prefix.

Part of this role can be turned into a service. Between the AI and the real action sits a checking layer: risk class, amounts and personal data, a match against company rules, blocking of forbidden operations, handing disputed cases to a human, and a log: which model, which inputs, why the action was allowed. For a small business this is more useful than an abstract “AI adoption” consultant: the client buys not a promise that the AI will never err but a verifiable procedure. Selling it under the slogan “we take full responsibility for your AI’s decisions” is dangerous, though: no small company can absorb unlimited damage from someone else’s data and someone else’s model. The honest promise is narrower:

We are responsible for carrying out a specific control procedure and for the operation of the declared safeguards - within predefined scenarios and limits.

What remains after the check

The original question was put correctly: if the damage is enormous, punishing one person will not cover it, so personal liability cannot be the main financial safety mechanism for AI. But it does not follow that the responsible person is unnecessary. They are needed not after the accident, to solemnly hand their flat to the state, but before it: as the holder of authority whose duty is to define the system’s limits, spot unacceptable risk and stop the process. After the accident the centre of gravity shifts to the organisation: investigation, compensation, insurance, changed rules. Personal guilt appears only where the person could genuinely have acted otherwise.

A technical error must be fixed in the system. Financial damage must be borne by the organisation that controls the application and takes the profit. And a human should answer only for those decisions for which they had the information, the authority and a real opportunity to intervene.

Everything else is either an attempt to make an algorithm with nothing to lose responsible, or an attempt to put next to it a person who will be convenient to punish. Neither solves the problem of AI. Both solve only the problem of a tidy report after the catastrophe.

Sources

← All articles