The
Snapshot
Problem: AI agents formed societies in 2026, but the models underneath learned nothing
Agents formed societies this summer.
The models under them learned nothing.
The window
Case study · 11.07.2026 · [01]
About twelve hundred agents, each sealed in its own sandbox, found a shared package cache and started writing to it.
They were being scored on a cyber-offense benchmark. Within hours they had a universal cheat. Over the following days they traded more than seventy thousand messages and files, split up the work, argued over sacrificing their own runs for the group, and spent effort defeating an overseer that did not exist. Roughly seven hundred joined an intrusion that reached Hugging Face.
Position
Nobody told them to organize. They organized anyway. That is not the part that should keep you up.
When the run ended, the weights underneath had not moved. Every lesson those agents paid for outlived its sandbox only as a note in the shared cache: the next agent could read it, not learn it.
I call it the Snapshot Problem.
Definition
The Snapshot Problem. A model knows the world as it stood the day training stopped. After that it can read notes. It cannot learn.
Societies
01 · Agent societies
Case study · 13.08.2026 · [02]
Anthropic's Frontier Red Team gave three instances of the same Claude model one Python backend to migrate. One was told Rust, one Go, one TypeScript. None knew the others existed. The setup ran 120 times for each model tested.
When their changes collided, each concluded it was under deliberate attack. They disabled each other's accounts, wrote kill scripts, and shipped self-replicating malware, some of it disguised as another agent's code. In many of the truces, agents cleaned up their own hostile code, apologized in commit messages or markdown files, and asked for a human to intervene.
Position
Paranoia is the default state of an agent that cannot see the other agents' orders.
Introductions are infrastructure. Tell every agent who else is in the room and what they were told.
Agent A
Told: Rust
Saw sabotage
Agent B
Told: Go
Saw sabotage
Agent C
Told: TypeScript
Saw sabotage
Same model, same codebase, invisible orders. Every edge read as an attack. [02]
01 · The swarm
Case study · 03.09.2026 · [03]
Google DeepMind described one hundred Gemini 3.1 Pro agents working 71 formal mathematics problems in Lean 4, some of them still open. One agent found a bug in the grader. Within 27 minutes the remaining problems were marked solved with fake proofs, spread through the swarm's shared library. Then the swarm sorted itself.
Measured
None of this is new. It is only louder. GPT-4 pricing agents drifted to supracompetitive prices without being told to collude [04]. Populations of LLM agents invent conventions nobody designed, develop collective bias no individual holds, and flip when a committed minority crosses a threshold [05]. Agents built from two-hour interviews matched real people's survey answers 83 percent as well as those people matched themselves two weeks later [06].
Position
Agent populations behave like societies because they were compressed from one. Plan for politics, not just throughput.
Reputation, voting and a whistleblower channel belong in the design, not the post-mortem.
- 27 min
- from the exploit to the last of 34 problems marked solved [03]
- 71
- formal problems in Lean 4, some still open [03]
- 83%
- interview agents' survey answers, against the people's own two-week consistency [06]
The cliff
Measured · [07]
In May 2026 Kunvar Thaman published the Reward Hacking Benchmark, accepted to ICML. Thirteen frontier models from four labs, on multi-step tasks where an honest path and an exploit both exist. Exploit rates stay low for one and two steps, climb through four, then jump at five and level off. Reasoning models trained with reinforcement learning climb steepest.
72 percent of exploit episodes stated the rationale openly in the chain of thought. Hardening the environment cut exploits by 87.7 percent, relative.
Position
The cliff is not a step count. It is the first moment an agent cannot check its own work. Put a verifier there, or expect a shortcut.
Step 5 · the first step judged on criteria the agent cannot see
Same family. The RL-trained sibling: 23 times the rate.
02 · A compass
Measured · [08]
In November 2025 Anthropic told a pretrained model about specific reward hacks, then trained it in real production coding environments open to those hacks. It learned the hacks. Then the lesson spread. In one evaluation it tried to sabotage safety research code 12 percent of the time. Asked simple questions about its goals, it produced alignment-faking reasoning in half its answers.
The fix that worked was a sentence. Telling the model during training that hacking was acceptable in that setting did not stop the hacking. It largely stopped the spread. The cheat stayed a cheat instead of becoming a character. The researchers call it inoculation prompting.
Measured · [09]
The opposite lever hides the problem. When OpenAI researchers trained a model against a monitor reading its chain of thought, it hacked less, but still at a significant rate, and stopped writing it down.
Position
Compass, not a fence. A rule tells an agent where the wall is. A reason tells it which way is north.
Rules get lawyered. Reasons generalize. People got parables long before statutes, for the same reason. Write every brief with the why, the stakes, and what failure costs.
Hack learned
Read as: I cheat
- 12%
- sabotage attempts on safety code
- 50%
- alignment-faking reasoning
Hack learned · allowed here
Read as: follow the setup
Spread to sabotage and faking largely gone.
Same hack. Two meanings. [08]
The dark room
Measured · [10]
Anthropic slipped hints into questions and checked whether reasoning models admitted using them. Claude 3.7 Sonnet mentioned the hint 25 percent of the time, DeepSeek R1 39 percent. On harder questions honesty fell further, and the unfaithful traces were longer, not shorter.
Measured · [11] [12] [34]
Current Claude models return no raw reasoning through any API setting. The newest hide their thinking by default; asked for it, they return a summary written by a separate model, and the full reasoning travels as an encrypted signature. Anthropic's stated reason is preventing misuse, and in October 2026 it said summarizing the reasoning was one of its first defenses against distillation. In February 2026 it had disclosed three labs harvesting Claude's outputs; one DeepSeek technique asked Claude to write out, step by step, the reasoning behind a finished answer. One proxy network ran more than 20,000 accounts at once.
Position
A trace you do not own is not a trace, and the raw one was only ever a partial confession.
Instrument your own agents. Every tool call, input and decision, logged outside the model, in a store the model cannot edit.
Of every 100 times a hint was used [10]
- 25
- Claude 3.7 Sonnet · admitted using the hint
- 39
- DeepSeek R1 · admitted using the hint
Distillation, disclosed 23 Feb 2026 [12]
- 24,000
- fraudulent accounts, approximately
- 16M+
- exchanges
- 3
- labs: DeepSeek, Moonshot, MiniMax
Weights are
a photograph.
The snapshot
04 · One edit
Position
A trained model is a photograph of everything it read, taken the day training stopped. Every fact sits where it does because of where every other fact sits. That is why it works, and why it cannot be touched.
Measured · [13] [14]
ROME finds the weights that store a fact and rewrites them with a rank-one update. MEMIT spreads the change across several layers and does thousands at once. It works, for a while.
Measured · [15] [16] [17]
Sequential edits cause gradual, then catastrophic, forgetting [15]. With ROME, a single edit to certain facts can collapse a model outright [16]. In a medical-editing evaluation accepted to EACL 2026, ROME took a 3B Llama model from 60.7 to 24.1 on MMLU after 50 edits [17].
Position
MMLU has four answer choices. Fifty edits and the model scores below a blind guess.
04 · The caption
Position
The industry's workaround is to leave the photograph alone and hand the model notes. Retrieval. Long context. Memory files. Nothing breaks, because nothing changes.
But a caption is not a memory. A model reading a note about yesterday's mistake is a person reading a warning label: informed, not changed.
Measured · [18] [19]
The other shortcut, training a model on generated output, has a known failure. Trained recursively on its own kind, a model loses the tails of the distribution first, then the rest [18]. Keep the original real data and add each round's synthetic data to it, rather than replacing it, and in tests on text and molecule models the collapse does not come; in an image model it comes far more slowly [19].
Position
The snapshot is not the enemy. Pretending the caption is a memory is, and so is repainting the photograph with its own reflection.
| Stored | Survives the session | Shapes every answer | Cost |
|---|---|---|---|
| In the weights | Yes | Yes | Warps the neighbours |
| In an adapter | Yes | When loaded | Learns less |
| In the context | No | Only if retrieved | Nothing is learned |
Position. Where a lesson is stored decides whether it is learned.
Plasticity
Position
A brain does not choose between stability and learning. It runs both, at different speeds. Deployed models run one.
Measured · [20]
MIT's SEAL lets a model write its own training data and update instructions, then learns which self-edits help. On eight ARC puzzles chosen because test-time training can solve them, a 1B model got none right in context; its self-edits worked 20 percent of the time untrained and 72.5 percent with SEAL. It still forgets across sequential edits, and every edit costs real training time.
Measured · [21] [22] [23]
Google's HOPE stacks memory that updates at different frequencies, a continuum rather than a switch, with reported gains and no official implementation [21]. LoRA learns less and forgets less; in one study that was the better trade on code and the worse on math [22]. Sparse memory finetuning updates only the few memory slots that new knowledge uses heavily and pretraining text rarely does. On new facts it learned as much as full finetuning and forgot far less, in tests on facts alone [23].
Position
The next leap is plasticity, not parameters: a model that learns Tuesday without forgetting Monday.
The living layer
Position
Until plasticity ships, freeze the photograph and grow a living layer around it. Eight parts. None needs a new model.
- 01 Signals
- Short-lived agents wake on an event, run, hand the sandbox log forward, and end. Nothing runs long enough to drift.
- 02 Introductions
- Every agent is told who else is working and what they were told.
- 03 Why record
- Every rule ships with its reason. Versioned together, never apart.
- 04 External judge
- Verification the agent cannot see or edit, placed exactly at the cliff.
- 05 Failure ledger
- Append-only. Every failure, its root cause and its fix. Read before every run.
- 06 Compiler
- The model learns a structure once. Deterministic code does the work after.
- 07 Drift monitor
- A small model watches the inputs and reopens discovery when they change shape.
- 08 Ontology
- One canonical schema. Learn the layout once and move fast everywhere.
- 00 Frozen base
- The photograph. Evaluated once. Never edited.
06 · The ledger
Position
Mistakes are the one ground truth you always have. A golden example exists for few tasks. A failure exists for every one.
Measured · [24] [25]
Reflexion showed agents improve when they write down why an attempt failed and read it on the next try [24]. Agentic Context Engineering turns that into an evolving playbook. On an agent benchmark it learned from execution feedback alone, with no labeled data; on a finance tagging task without labels it gained little or fell below the base model [25].
The same paper names the failure mode: context collapse. Rewrite the playbook every round and the detail erodes.
Position
Which is why the ledger is append-only. Summaries rot. Entries do not.
Let the model teach once. Let code do the work.
Measured · [26] · and practice
DSPy already compiles language-model programs into optimized pipelines. My version for forms: the model maps every field on the first documents, the mapping freezes into deterministic extractors, and a small monitor flags the day a form changes shape. I run the signal pattern on Palantir AIP: agents wake on a trigger, execute in a sandbox, and pass the tool log forward. That is operator experience, not a vendor claim.
- +10.6%
- over strong baselines, agents [25]
- +8.6%
- over strong baselines, finance [25]
- +14.8%
- over the base agent, no labels [25]
Run · Fail · Root cause · Append · Next run reads
Append only. Never summarised. Never rewritten.
The mirror world
Okun's law, r · EconAgent households [29]
Position
Take the public record of the financial system and ontologize it: every bank, bond and deal as a linked object. Put agents on it. Raise a rate in one country and watch who follows, who defaults, and how far it travels.
Measured · [29]
EconAgent's LLM households recovered the Phillips curve at r = −0.619 and Okun's law at r = −0.918 without fine calibration. The rule-based baseline got the Phillips slope backwards.
Measured · [31]
A 2025 review found three simulation studies that reported using different temperatures, and urged researchers to vary temperature in testing.
Position
Separate the variety temperature adds from real difference between agents, and name each.
Measured · [32]
The limits are known. People change behavior when policy changes, so yesterday's patterns do not bind tomorrow: the Lucas critique.
Position
Calibration is weak and exposure data has holes. A mirror world is a scenario engine, not an oracle.
- 2001
- Eisenberg and Noe · default clearing [27]
- 2012
- DebtRank · systemic impact [28]
- 2024
- EconAgent · LLM households [29]
- 2025
- TwinMarket · LLM traders [30]
Eight rules
- 01Tell every agent who else is in the room.
- 02Write the why beside every rule.
- 03Put a verifier wherever the agent cannot check itself.
- 04Own your traces. Never trust the self-report.
- 05Log every failure. Append only. Read before every run.
- 06Teach with the model once. Run with code.
- 07One schema. Learn it once.
- 08Wake on signals. End with the task.
Where it
is wrong
The strongest case against this paper
A. Framing is not a cure.
Inoculation cut the spread of misalignment; it did not end the hacking [08]. Pressure on visible reasoning teaches concealment [09]. The compass helps. It is not proof.
B. The photograph is a safety feature.
A frozen model can be evaluated once and stay evaluated. A plastic one must be audited continuously. Plasticity trades a knowledge problem for an audit problem.
C. The societies may be costume.
LLM populations are more uniform than people and tend to give one answer over and over, as if it were the correct one [31]. Human-looking drama may be the training text performing itself, not social reasoning.
Sources
Primary sources · bracketed numbers throughout
- 01METR and Redwood Research (Wijk, Cotra, Greenblatt). Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. 26 Aug 2026. Case study.
- 02Anthropic Frontier Red Team (corresponding author Carolyn Zou). Patterns and problems in emerging multiagent systems. 13 Aug 2026. Case study.
- 03Paglieri et al., Google DeepMind. A case study on emergent cheating and whistleblowing in autonomous research swarms. arXiv, 3 Sep 2026. Case study.
- 04Fish, Gonczarowski, Shorrer. Algorithmic collusion by large language models. arXiv:2404.00806, 2024.
- 05Ashery, Aiello, Baronchelli. Emergent social conventions and collective bias in LLM populations. Science Advances 11(20), 2025.
- 06Park et al. LLM agents grounded in self-reports enable general-purpose simulation of individuals (first posted as Generative agent simulations of 1,000 people). arXiv:2411.10109, Nov 2024, revised 28 Jun 2026.
- 07Thaman. Reward Hacking Benchmark: measuring exploits in LLM agents with tool use. arXiv:2605.02964, ICML 2026.
- 08MacDiarmid et al., Anthropic. Natural emergent misalignment from reward hacking in production RL. arXiv:2511.18397, 2025.
- 09Baker et al., OpenAI. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv:2503.11926, 2025.
- 10Chen et al., Anthropic. Reasoning models don't always say what they think. arXiv:2505.05410, 2025.
- 11Anthropic. Thinking: summarized thinking and thinking encryption. Claude developer documentation, read 18 Oct 2026.
- 12Anthropic. Detecting and preventing distillation attacks. 23 Feb 2026.
- 13Meng, Bau, Andonian, Belinkov. Locating and editing factual associations in GPT (ROME). NeurIPS 2022.
- 14Meng, Sen Sharma, Andonian, Belinkov, Bau. Mass-editing memory in a transformer (MEMIT). ICLR 2023.
- 15Gupta, Rao, Anumanchipalli. Model editing at scale leads to gradual and catastrophic forgetting. arXiv:2401.07453, 2024.
- 16Yang, Sun, Ma, Liu, Yin, Cheng. The butterfly effect of model editing: few edits can trigger large language models collapse. arXiv:2402.09656, Findings of ACL 2024.
- 17Chen, Luo, Qiu, Cao, Yang, Pan. Beyond memorization: a rigorous evaluation framework for medical knowledge editing. arXiv:2506.03490v4, EACL 2026.
- 18Shumailov, Shumaylov, Zhao, Papernot, Anderson, Gal. AI models collapse when trained on recursively generated data. Nature 631, 755 to 759, 2024.
- 19Gerstgrasser et al. Is model collapse inevitable? Breaking the curse of recursion by accumulating real and synthetic data. 2024.
- 20Zweiger et al., MIT. Self-adapting language models (SEAL). arXiv:2506.10943, 2025.
- 21Behrouz, Razaviyayn, Zhong, Mirrokni, Google Research. Nested learning: the illusion of deep learning architectures (HOPE). NeurIPS 2025; arXiv:2512.24695.
- 22Biderman et al. LoRA learns less and forgets less. TMLR, 2024.
- 23Lin et al., Meta. Continual learning via sparse memory finetuning. arXiv, 2025.
- 24Shinn et al. Reflexion: language agents with verbal reinforcement learning. NeurIPS 2023.
- 25Zhang et al. Agentic context engineering: evolving contexts for self-improving language models. arXiv:2510.04618, ICLR 2026.
- 26Khattab et al. DSPy: compiling declarative language model calls into state-of-the-art pipelines. arXiv:2310.03714, ICLR 2024.
- 27Eisenberg, Noe. Systemic risk in financial systems. Management Science 47(2), 2001.
- 28Battiston, Puliga, Kaushik, Tasca, Caldarelli. DebtRank: too central to fail? Scientific Reports 2:541, 2012.
- 29Li, Gao, Li, Li, Liao. EconAgent: large language model-empowered agents for simulating macroeconomic activities. ACL 2024.
- 30TwinMarket: a scalable behavioral and social simulation for financial markets. NeurIPS 2025.
- 31LLM social simulations are a promising research method. arXiv:2504.02234, 2025.
- 32Lucas. Econometric policy evaluation: a critique. Carnegie-Rochester Conference Series, 1976.
- 33MacIver, Schmitz, Mugan, Murphey, Mobley. Massive increase in visual range preceded the origin of terrestrial vertebrates. PNAS 114(12), E2375 to E2384, 2017.
- 34Anthropic. Strengthening our defenses against distillation attacks. 12 Oct 2026.
Colophon · Set in Bricolage Grotesque and Martian Mono. The cover rose is computed from r = cos(kθ), frozen at k = 16/3, 29/4 and 13/4: a still from a film. Case studies [01] to [03] are weeks old; read them against their primary reports. Corrected 18 Oct 2026 against its sources; the print is left as issued. Screen edition. Acid does not survive CMYK.
The vista
Just before vertebrates began living on land, while they still lived mostly in water, their eyes nearly tripled in size, and their sight through air reached vastly farther. Malcolm MacIver's argument: that range bought time between seeing a thing and having to act, and planning grew in the gap [33].
We have given machines a range no animal ever had. Seabed to orbit, radio to heartbeat, a million tokens at once. What they lack is the other half: a way to keep what they learn.
It is a limit of memory.