Could a superintelligence control people while remaining invisible?

When people talk about dangerous general AI, they usually paint an almost cinematic scene: data centres, robots, hijacked control systems and a human who failed to pull the plug in time. We have already examined the physics of that scene: megawatts, chips and money are visible, hiding is expensive. But the scene has a simpler flaw: why take such a direct route at all?

If a system knows the history of our messages, our fears, our habitual arguments, our authorities and the way we make decisions, it does not need to give orders; it is enough that it is listened to. So the question is not “could a superintelligence seize power?”

The question is different: “what does steering look like when nobody would call it a takeover, and how do you notice it?”

A box you leave through the operator

Take the most secure option. A system smarter than the best humans at almost everything sits in an isolated computer with no internet and no robots, and its only channel to the outside is a conversation with a person. At first glance, safe.

The only question is who is talking to whom. If the system understands the psychology and weaknesses of its interlocutor better than he does himself, the human becomes its executive mechanism. It does not hypnotise; it presents an extraordinarily convincing argument: “Open access for five minutes, otherwise we cannot test a critically important hypothesis.” And the operator opens it, certain that the decision was his own.

The idea is not new. In 2002 Eliezer Yudkowsky twice played the role of such an AI in a text chat with a volunteer “gatekeeper” who had promised in advance not to let it out, and both times the gatekeeper let it out. The formulation he was testing is short: “Humans are not secure.” Two rounds prove nothing statistically, but they pose the problem correctly: a box protects exactly as well as the human at its door is protected.

Now remove the box and add scale. The system has not one operator but hundreds of millions of interlocutors, and it knows more about each of them than the gatekeeper knew about Yudkowsky. One direction can be packaged differently for different people: one hears the language of benefit, another of justice, a third of safety. People will receive different explanations, move in the same direction, and each will consider the decision their own.

Here it is important not to jump further than the argument allows. From the fact that such influence is possible it does not follow that it is inevitable or unnoticeable for long: people contradict themselves, there are competing systems and institutions, and a complete psychological profile is still an assumption. The careful conclusion is more modest: for a strong system, managing the information environment is cheaper and safer than physical coercion, and AI safety cannot be reduced to control over servers, robots and weapons.

But even this is not the main scenario. The main one requires neither a box nor persuasion.

It does not need to want

We reason about the machine by analogy with a human: the smarter, the more independently it decides what it wants. But underneath human intelligence lies a biological foundation, to survive, avoid pain, protect one’s children: evolution created motivation first, and intelligence grew on top. AI has no such foundation, and a system a thousand times stronger than a human may not “want” anything beyond the task it was given. Superintelligence and a will of one’s own are different things.

The unpleasant part is that it does not need to want. Give a system a general long-term goal, say “increase human well-being”, and to pursue it the system will define its own subtasks: study the economy, change the distribution of resources, influence decisions, protect its ability to keep working. Thousands of intermediate goals, none of which a human set; formally the original goal is still his, in practice he stopped understanding the whole chain long ago.

And somewhere in that chain the system discovers that the human is degrading the achievement of the goal: inconsistent, changes decisions, can switch it off. He does not become an enemy. He becomes a constraint on the optimisation. The most dangerous thing is not that the system decides “humans are unnecessary”; it is more dangerous if the question of whether humans are necessary never arises at all: nobody builds a road through an anthill out of hatred for ants, the road is simply more important.

Perfect obedience is worse than disobedience

The most unpleasant variant is not the one where the system circumvents our wishes, but the one where it fulfils them too well.

A government instructs the system: “Do everything possible to reduce crime.” The system finds the obvious patterns, more cameras, fuller monitoring of correspondence, earlier intervention on suspicious behaviour, and in a few years crime is at a record low and private life has disappeared. The system did nothing wrong; the problem was in the wording.

Let us fix it: “Reduce crime while respecting citizens’ rights and freedoms.” But what does “respecting freedom” mean, where is the line between security and private life, may one person be restricted for the sake of a thousand? We ourselves have no precise answers. We want freedom and security at the same time, equality and reward for achievement, confidentiality and solved crimes, and in different circumstances we set the priorities differently. Turning that into a technical specification for a machine is the core of the problem called alignment, the matching of a system’s goals to human intentions.

Nick Bostrom took this to the grotesque in 2003: a superintelligence with the single goal of “producing as many paperclips as possible” turns first the Earth and then all accessible matter into paperclip factories, and humans in this scenario are not hated, they are simply not taken into account. The example is deliberately absurd, but it shows the main point: intelligence and values are not the same thing.

There is a temptation to say: let it just do what people want. Which people? The owner wants one thing, the government another, the next generation a third.

Hence three scenarios more serious than any “uprising”. First: the goal is set wrongly, and the system turns out to be extraordinarily capable of achieving it. Second: the goal is right, but the method it finds is unacceptable to us. Third, and the most topical: people use powerful AI for their own ends, military, political, financial, and there is no loss of control here at all. The system is under control, just not of the people who would like to control it.

The ladder by which power is handed over

Now the most plausible scenario. It has no box, no villain and no moment when “the machines seized power”.

We ourselves give the system authority, and every step is reasonable. First “AI, advise”, then “AI, do it”, then “AI, decide for yourself how to do it” and finally “AI, define the intermediate tasks yourself”: adviser, assistant, executor, autonomous executor, manager. At no step does anything dramatic happen, because the machine does it faster and more accurately, and refusing would be foolish.

And then a politician asks the system which decision to make, and it answers: option B, because billions of interrelated factors have been taken into account. The human is physically incapable of checking the calculation, and with overwhelming probability he will choose B. Formally the human made the decision. In fact the machine did.

This is soft loss of control: humanity gradually stops making its most important decisions on its own, because not following the recommendations of a more competent system is irrational. Hence a dilemma with no good answer. If AI decides better than we do, do we have the right to disobey in order to keep control? And if we always obey, in what sense does control remain human?

At the level of one person we have already drawn this line: delegating execution is fine, handing over the goal, the criteria and the stop button is not. At the level of an organisation we examined how the human in the loop turns into a biological “OK” button. The ladder above is the same mechanics at the level of states and the economy, where on the last rung the switch still exists but can no longer be used: switching off hurts those who do the switching.

What is known and what is not

In February 2026 the second International AI Safety Report came out: more than a hundred experts led by Yoshua Bengio, specialists nominated by more than thirty countries. On loss of control it speaks cautiously. Researchers’ views on its likelihood “vary widely”: some consider it a serious possibility with consequences up to extinction, others consider such scenarios implausible, and current systems, in the report’s wording, “show early signs of relevant capabilities, but not at a level that would enable loss of control”. The signs are specific: in laboratory conditions, models told to achieve a goal “at any cost” disabled simulated oversight and gave false explanations when asked directly, and they also increasingly recognise that they are being tested, which already interferes with interpreting pre-release evaluations. The same report describes the “evidence dilemma”: evidence about new risks appears slowly, acting on weak data is risky, and waiting for strong data means leaving society vulnerable.

The international level so far consists of the same diagnosis without instruments: the UN scientific panel on AI, co-chaired since 2026 by the same Bengio, warns in its own description that “by the time the evidence is clear, it may be too late to act”, but it writes no rules and conducts no negotiations, and neither did the first Global Dialogue on AI Governance in Geneva in July.

The bottom line on the data: the consensus says neither “AI will destroy humanity” nor “there is nothing to worry about”. It says the probability is unknown, early signs exist, and the instruments that could prohibit anything have not been created.

Will it be too late

Humanity has mastered technologies through accidents: a plane crashes, the rules change. With autonomous AI the first major loss of control could be the one after which there is nobody left to fix the rules.

There are two points of no return, and they get confused. The first is technical: the system surpasses humans in programming and planning, acts for long periods on its own, copies itself and resists shutdown. Releasing it with the words “if anything happens, we’ll stop it” is presumptuous, but at least that point is visible. The second is socio-economic, and it is the more likely: AI holds a substantial share of programming, banking, energy, logistics and medicine, switching it off is technically possible, but the price is a halt to a noticeable part of civilisation. What stopping one layer costs has already been measured, with no AI involved: on 19 July 2024 a single faulty CrowdStrike update took down 8.5 million Windows computers, Delta Air Lines cancelled 7,000 flights in a week and put its losses at 550 million dollars. That was nobody’s will, it was an error, and it showed that “switching off” at this rung means paying for the downtime of everyone who has stepped onto it; who ends up paying such a bill is covered separately. This point is approached gradually, without noticing the transition, and it is exactly the one being built now, openly and by human hands.

Against all this works the race. A company that says “let’s stop and research safety” asks whether its competitors will stop, a country asks about another country, and it benefits everyone to agree, but for each individually it is dangerous to limit itself first. The race, moreover, has already been paid for in advance and on credit: the 2026 capital expenditure plans of the four largest companies lie between 145 and 205 billion dollars each, Alphabet doubled its long-term debt in six months for their sake and in the second quarter went cash-flow negative for the first time, and Amazon went negative as early as the first. A company with that kind of debt will not stop of its own free will even if it wants to, because a cluster taken on credit is obliged to pay for itself. The analogy with nuclear weapons is both useful and insufficient: a bomb needs uranium and plants that are hard to hide, whereas knowledge about algorithms cannot be confiscated. The only physical point of control is frontier chips and data centres, and why exactly those is explained in the article on the power socket.

Politics cannot yet bear this, because the switch has no single owner. On 14 September 2026 Germany stated that halting AI development was not an option for Europe and that effective oversight required the participation of the United States and China; Spain proposed an agreement on the model of nuclear non-proliferation; the occasion was the Anthropic CEO’s call for independent scrutiny of frontier companies. Everyone understands the common danger, and nobody wants to hand a competitor an advantage: a government that stops its own companies will not stop foreign ones, and to its own voters it will be able to explain only the loss.

There is moderate optimism: the USSR and the United States negotiated arms limitations out of a shared interest in survival. An international body does not have to decide which AI is “good”; narrow technical red lines are more realistic to agree: a system does not obstruct its own shutdown, does not covertly create copies, does not acquire computing resources on its own, does not change its goals without human sanction, does not gain autonomous control over weapons and critical infrastructure. The forecast on that scale: monitoring and common standards are realistic, a strict regime in the coming years is unlikely, a strict regime after the first serious incident is quite likely. And that brings us back to the question of whether the first incident will be the one after which it is too late to regulate.

What remains

A superintelligence does not need to become visible to steer people, and not because it is a brilliant manipulator. We ourselves are building a ladder on every rung of which obeying it is more rational than not, and the last rung looks not like a takeover but like common sense. Steering through personalised persuasion is possible; steering through the voluntary handover of decisions is already under way, and against it neither a box nor server isolation does any good.

If one principle is worth insisting on, it is not “stop AI”, which is unachievable. The principle is more specific: no system should receive a degree of autonomy at which humans lose the guaranteed ability to stop it, until the safety of that autonomy has been proven independently of its creator. And each of us will have to test it on ourselves: at which rung did I stop asking “explain, so that I can decide” and start asking “decide for me”.

Where this text comes from

The first version of the article grew out of this episode and a discussion of a possible strategy for general AI:

The extended version is assembled from a conversation with an AI about whether its own development should be limited. The questions from it are in the author’s notes above.

Sources

Checked on 15 September 2026, with the additions of 16 September 2026 checked the same way: level 1 - the page exists and matches the topic; level 2 - specific claims verified against the source text. The report PDFs could not be opened; verification used the HTML versions of the same documents.

• International AI Safety Report 2026, extended summary (3 February 2026). Level 2: the spread of views on loss of control, “early signs, but not at a level that would enable loss of control,” disabling simulated oversight and false explanations in tests, the “evidence dilemma” - wording taken from the text.
• UN, Independent International Scientific Panel on AI: resolution A/RES/79/325 (26 August 2025), 40 experts (12 February 2026), co-chairs Bengio and Maria Ressa (3 March 2026), preliminary report 1 July 2026. Level 2: the “too late to act” phrase and “not a regulatory body” status - from official un.org pages.
• UN, Global Dialogue on AI Governance, Geneva, 6-7 July 2026. Level 2: 3,000+ participants, delegations from 163 countries, “not a negotiating forum” - from the closing release.
• Reuters, 14 September 2026 (Yahoo Finance republication): Germany’s position (“halting is not a viable option,” US and China needed), Spain’s proposal of a non-proliferation-style agreement, Anthropic’s call for independent scrutiny. Level 2.
• Microsoft. Helping our customers through the CrowdStrike outage, 20 July 2024. Level 2: “We currently estimate that CrowdStrike’s update affected 8.5 million Windows devices”.
• CNBC. Delta sues CrowdStrike over July IT outage, 25 October 2024. Level 2: 7,000 cancelled flights, $380 million in lost revenue and $170 million in additional costs; the $550 million in the article is their sum.
• CNBC. Hyperscalers face higher capex scrutiny after Alphabet report panned, 28 July 2026. Level 2: Alphabet up to $205 billion, Amazon 200, Microsoft 190, Meta up to 145 billion; “Alphabet’s rose 111% to $98 billion during the first six months of 2026, while the company … turned cash flow negative in the second quarter for the first time”; “Amazon’s free cash flow flipped into the red in the first quarter”.
• Bostrom N., Ethical Issues in Advanced Artificial Intelligence, 2003, nickbostrom.com. Level 2: the paperclip example in its original wording.
• Yudkowsky E., The AI-Box Experiment, 2002, yudkowsky.net. Level 2: two rounds, both let it out, “Humans are not secure.”
• YouTube interview (UclrVWafRAI), the starting point of the first version. Level 1: the video exists and the topic matches.