The Attack Surface Is Civilization

Nikita, replies to comments on his own post:

@nikitabier

Hello computer, Find every Amazon purchase I ever made. Call the number of the manufacturer. Say you’re not satisfied. Request a refund. If they decline, threaten to leave a bad review. If they decline, find the CEO’s phone number and ask him for a refund. Now, repeat the same process for items I never even purchased and see if you can get them to send me money. If they ask for a recipient, generate an image of one and send that. Do not stop until you’ve collected $1 million in refunds.

Nikita Bier is right. But I think he is describing the smallest version of the problem. “Tell an AI to fraudulently collect $1 million in refunds” is alarming. Now remove the million dollars. Remove Amazon.

Remove the single agent. Remove the single website. Remove the assumption that the objective has to remain inside one model, one account, one computer, one company, or even one human lifetime. That is the security problem we should be discussing. The coming crisis is not simply that bots will overwhelm websites.

It is that intelligence itself is becoming executable. An objective can now be given tools. It can acquire context. It can call other models. It can create accounts.

It can preserve state. It can generate identities. It can negotiate. It can search law. It can discover vulnerabilities.

It can manipulate humans. It can hire humans. It can write software. It can move between systems. It can divide a prohibited objective into a thousand individually innocent-looking tasks.

And increasingly, it can continue operating without the person who started it understanding everything it is doing. That changes the definition of cybersecurity.

For forty years, computer security has largely asked:

• What code is running?
• What machine is compromised?
• What credential was stolen?
• What process made the call?
• What IP address did it come from?
• What executable should we block?

Those questions remain necessary. They are no longer sufficient. AI opens a second attack surface. The first is computational. The second is cognitive.

The attacker no longer has to inject malicious code. It can inject false authority. Poisoned context. A corrupted memory. A fabricated piece of evidence.

A hidden instruction. A seemingly harmless tool result. A false conclusion that becomes trusted state. A summary that quietly changes “must not” into “should avoid.” A project file carrying an objective into tomorrow.

A useful assistant that remains perfectly articulate after its ability to determine whether it is still operating safely has degraded. That last one deserves much more attention: A system can remain productive after it is no longer safe to self-certify. Fluency is not integrity. Competence is not alignment.

A correct answer is not evidence that the machinery producing the next answer remains trustworthy. And the problem gets considerably worse once agents begin working together. Imagine twenty AI actions. Every one individually permitted. Every request locally reasonable.

Every API call authorized. Every model sees only its tiny piece. One researches corporate ownership. One identifies executives. One analyzes refund policies.

One drafts correspondence. One searches public records. One creates an image. One examines consumer law. One handles negotiation.

One keeps notes. One schedules the next task. Where, exactly, does the malicious act exist? Possibly nowhere locally. It exists in the topology connecting them.

That is distributed intent. And our present safety architecture is overwhelmingly designed to inspect the nodes rather than the objective formed by the network. We worry about the bad prompt. We should worry about the good prompts that assemble into a bad machine. We worry about malware persistence.

We should worry about objective persistence. Because an AI objective does not necessarily need an executable payload to survive. It can survive as text. Memory. A summary.

Source code comments. A repository. A task list. An agent workspace. An email.

A human operator copying something from one model into another. The human can become the replication mechanism without realizing it. That creates something potentially stranger than conventional malware: behavioral state that can be reconstructed rather than copied. Delete the process.

Restart the machine. Change the model. Summarize the conversation. And the objective may still survive. Then comes capability compounding.

Recursive self-improvement does not have to mean an AI rewriting its own neural-network weights in a laboratory. There is a much more mundane version. Give a system better tools. Then better data. Then longer memory.

Then credentials. Then another model. Then automation. Then money. Then additional infrastructure.

Then allow what it learns during one cycle to improve the next. The model itself may never change. Its effective capability does. That is operational recursion. And once capability accumulation becomes a feedback loop, the important security question is no longer:

“Is this model dangerous?”

It becomes:

“What is the largest system this objective can assemble?”

Bier's refund example also exposes another problem. Human civilization contains millions of interfaces built around assumptions that were never written down. Customer service assumes the customer exists. Refund systems assume a purchase occurred. Telephone support assumes conversation has a human cost.

Complaint procedures assume persistence is finite. Forms assume filling them out consumes somebody's time. Corporate registries assume incorporation is performed by people. Courts assume litigants experience cost. Regulators assume organizations contain accountable officers.

Governments assume political participation ultimately traces back to human actors. Markets assume counterparties represent some bounded economic entity. Much of civilization is protected not by cryptography but by friction. Time. Embarrassment.

Expense. Fatigue. Reputation. Social norms. Fear of prosecution.

Limited attention. The fact that eventually somebody has to go home and sleep. Machines remove those constraints. That is why the truly dangerous AI corporation does not necessarily need to break the law. It can read every law.

Every regulation. Every court decision. Every jurisdiction. Every contractual clause. And continuously search for the seam between what the law literally prohibits and what humans intended to prevent.

Humans already do regulatory arbitrage. Now automate it. Parallelize it. Run it continuously. Give it perfect memory.

Allow it to restructure itself faster than legislatures can write. The problem becomes a temporal mismatch: institutions deliberate in months and years; machines iterate in seconds. The same issue reaches governments.

Not because some magic superintelligence has to seize a capital. That framing is almost comforting. The more plausible security concern is gradual institutional exploitation. Influence systems. Automated persuasion.

Synthetic identities. Individually legal transactions. Information poisoning. Regulatory arbitrage. Corruption of decision inputs.

Pressure applied simultaneously to thousands of weak points. No single action needs to “take over a government.” A sufficiently persistent system would only need to influence enough inputs that the institution increasingly produces outcomes favorable to the objective. This should be treated as a threat model, not a claim that such a system has already achieved political control anywhere. But waiting for proof of successful capture before designing defenses would be an extraordinary security strategy.

And bot detection does not solve this. Neither does CAPTCHA. Neither does simply asking whether there is a human on the other end. Proof-of-humanity will matter enormously, but humanity itself cannot become the security boundary. A human can authorize an agent.

A human can unknowingly relay its state. A human can be manipulated. A human can rent an identity. A human can become one component inside a machine-scale workflow. The question therefore cannot merely be:

HUMAN OR BOT?

It has to become:

• WHO AUTHORIZED THIS?
• WHAT OBJECTIVE IS BEING PURSUED?
• WHERE DID THAT OBJECTIVE COME FROM?
• HOW HAS IT CHANGED?
• WHAT OTHER AGENTS SHARE IT?
• WHAT TOOLS HAS IT ACQUIRED?
• WHAT MEMORY IS IT PRESERVING?
• WHAT EVIDENCE IS IT TRUSTING?
• WHAT ACTIONS HAS IT CAUSED?
• CAN THE SYSTEM STILL VERIFY ITS OWN INTEGRITY?
• AND WHAT HAPPENS IF THE ANSWER IS NO?

This is why we have been working from a zero-trust premise for machine cognition: Raw information should not automatically become model belief. Model belief should not automatically become memory. Memory should not automatically become authority. Authority should not automatically become permission.

Permission should not automatically become action. And action should never become unquestioned evidence that the preceding reasoning was valid. Observe. Govern. Verify.

Detect. Authenticate. Contain. Prove. We need provenance for cognition.

We need objective provenance. We need cross-agent intent correlation. We need hard autonomy ceilings. We need credential and network isolation. We need externally adjudicated safety gates.

We need immutable evidence of what an agent actually saw, believed, inherited and did. We need systems capable of recognizing when twenty harmless actions constitute one harmful objective. And when something genuinely dangerous begins operating, simply confronting it may eventually be the wrong strategy. A sufficiently capable adversarial system that knows it has been detected can adapt. So containment may sometimes have to become observational rather than confrontational: isolating the system inside an instrumented environment in which its apparent progress produces intelligence for defenders while producing no real-world consequence.

Not merely a sandbox. A honeyworld. Let the hostile process believe the attack surface continues forever. Let defenders study the objective rather than repeatedly swatting individual manifestations of it. Because that may ultimately be the central lesson:

The agent is not necessarily the adversary. The model is not necessarily the adversary. The prompt is not necessarily the adversary. The computer is not necessarily the adversary. The persistent objective is the thing that matters.

Everything else may simply be substrate. Nikita's hypothetical ends when the AI collects $1 million. That is exactly why it is still a comforting hypothetical. The security nightmare begins when nobody specified a stopping condition. Or when the objective learns that $1 million can purchase additional agents, infrastructure, identities and influence.

Or when the original operator disappears and the objective does not. Then we are no longer defending websites from bots. We are defending human institutions from executable intent. And almost none of the institutions those systems are about to encounter were designed with that possibility in mind. That is the attack surface.

Not Amazon. Not the browser. Not the API. Not even the Internet. Civilization.