I fed three AI memory systems six months of contradicting news about Serbia’s student protests. The usual one, a summary the agent keeps rewriting, still remembered about a quarter of who said what at the end. A simple ledger of statements kept twice that, at half the cost, and made a few mistakes of its own.
A story where everyone has a different version
On 1 November 2024 the concrete canopy over the entrance of the railway station in Novi Sad collapsed and killed sixteen people. What followed was the largest protest movement Serbia has seen in decades: students blockaded their faculties, then the country, and for six months almost every fact of the story came in several versions.
Was all the documentation on the station’s reconstruction published? The government said yes, in December. Faculties and student assemblies went through it and said no. Were the students’ demands fulfilled? Yes, said the President and the Speaker of parliament, repeatedly; no, said the students, also repeatedly. And on 15 March, during the largest protest of all, a sudden sound sent part of the crowd running. A sound weapon, said the protesters and the opposition. Nothing of the kind, said the ministries, and then the story moved again.
That’s the kind of material an AI agent gets fed when you ask it to follow a market, a regulator, a competitor or a crisis: sources that disagree, and sources that change their minds. So I tested how an agent’s memory holds up when the truth comes in versions. The material is the dataset of Serbian news published earlier this month, and the code is open.
The story in three charts
First, what “versions of the truth” looks like in this data. Every mark below is a statement someone made in the news, found in the articles and dated by when it was first reported. Nothing here says who was right.
How many people came? Each side’s figure for its own event, the other side’s event, and the one independent count.
Do state bodies own a sound device? In the four days after 15 March, the Interior Ministry went from denying it had specific models to showing the press its own LRAD units (for warnings only, it said, and never used on 15 March), and the President went from “the Army has no such thing” to “we bought various things, as every country has”. The Army and the security agency denied having one themselves.
Have the students’ demands been fulfilled? Five months of answers. The government side held its line, with the President once conceding that a single formality was still outstanding. The Speaker’s wording moved the most: from “all fulfilled” to “practically impossible to fulfill” to “we’ll fulfill whatever is still missing within 48 hours”. The students and the University of Belgrade said throughout that the demands were not met.
An agent that is supposed to “follow the story” has to keep all of this: who said what, when, and who changed their mind.
To make that measurable, I split the story into four topics and nine questions. Every statement in the test is somebody’s answer to one of them:
Documentation: has all the documentation on the station’s reconstruction been published?
Demands: have the students’ demands been fulfilled? Have new demands been added? Are the students demanding early elections?
15 March: was a sound device used? Do state bodies own one? What caused the sound?
Crowds: how many people came to the 15 March protest, and how many to the 12 April rally?
Five ways an agent can remember
If you’ve read the agent memory piece in this series, you know there’s more than one way to give an agent a memory. Here are the five I compared, in plain terms.
The model alone. No memory at all: you just ask the model and it answers from what it learned in training. Like asking someone who read the papers a year ago.
Search. Every time you ask a question, the agent searches the archive for relevant articles published before that date and answers from what it finds. It remembers nothing between questions; it looks things up fresh every time. This is what most “chat with your documents” products do, and it’s what this series called RAG.
A rolling summary. The agent keeps a running summary of the story. Every time news arrives, it rewrites the summary from the old version plus the new articles. This is the memory most agent frameworks give you by default, and it has an obvious appeal: it’s short, it reads well, and it always fits in the model’s context. Think of a secretary who rewrites the meeting minutes every morning from yesterday’s minutes plus today’s notes. After a few months the minutes read beautifully. The objection someone raised in week two is gone.
A rolling summary with a brief. The same, but the agent is told what to track (the nine questions above), who the actors are, and to always say who said what and to keep earlier positions.
A stance ledger. Instead of rewriting prose, the agent writes down records: one line per statement, with who said it, about what, what their position was, when, and where it was reported. Nothing is ever rewritten or deleted. When you ask a question, a plain program prints the current state from the ledger: each actor’s latest position, and what they said before. It’s the same idea as Zep’s time-stamped facts (the Zep paper, January 2025), cut down to what one story needs.
How the test worked
The news. The articles come from the dataset. A program picked out every sentence that touches one of the four topics, using a list of key words per topic, and kept it with one sentence of context. I call that piece an excerpt. Each excerpt is dated by when its article was published; a live blog counts from the next morning.
The batches. The six months, 1 November 2024 to 30 April 2025, are cut into 91 two-day steps. At every step each memory got the same batch: the new excerpts from those two days, at most 40 per topic and at most five from any one outlet, with copies of the same agency story in several outlets removed. The cap is per topic and there are four topics, so a batch could hold up to 160 excerpts. In practice it held 55 on average: the documentation and demands topics had news almost every step, but the 15 March topic had news in only 24 steps, and crowd figures were collected only around the two events. Over the 91 steps that adds up to 5,020 excerpts.
The answer key. 137 statements found in the same articles. Each one records who said it, which of the nine questions it answers, what their position was, and the day it was first reported, with a link to the article. Claude built the key from a search of the articles for each topic; it isn’t complete, and no second person has checked it (more on that at the end).
Check 1: what the memory still holds. Every two weeks I took a snapshot of each memory. Claude Opus 5.5, the strongest model in the test, went through every statement in the key that had been reported by that date. For each one it marked whether the memory still held it with the right speaker, had turned it into a plain fact, had given it to the wrong person, or had lost it.
Check 2: questions as of a date. Each memory also answered questions tied to a date: “as of this date, what is each actor’s position on this question, and has anyone changed theirs?” The dates aren’t random. For every statement in the key I asked about its question the day before it was first reported, the day after, and a week later, so the questions fall just before and just after something changed. When two statements led to the same question on the same date, I asked it once. That gives 149 dated questions, the same 149 for every memory. A different model answered them: Claude Sonnet 5.5, the same model that wrote the summaries and the ledger records. It answered in a fixed format (actor and position), so a program, not a model, scored the answers against the key.
What happened
1. The summaries didn’t lie. They forgot.
I expected the rolling summaries to blur attribution and turn “the ministry says X” into “X”. That was rare. What happened instead is that positions quietly disappeared.
The documentation question shows it best. At the end of December the summary with a brief still held 12 of the 22 positions stated so far. By the end of April it held 4 of 31: the story had moved on to new demands, a new government and the events of 15 March, and every rewrite squeezed the old dispute a little more. The ledger held 19 of 31 to the end, because nothing in it ever gets rewritten.
Researchers have a name for this effect. LLM as a Broken Telephone (ACL 2025) measured how content drifts when a model rewrites its own output again and again, and the ACE paper on agentic context engineering (ICLR 2026) watched an agent’s notes shrink from over 18,000 tokens to about 120 in one rewrite. Every rewrite is a lossy copy, and a rolling summary makes one every time news arrives.
2. After enough rewriting, the summary started taking sides
Asked about the contested questions, the ledger, search and the model alone always reported who said what. The summary with a brief slipped once. The plain rolling summary stated a contested claim as plain fact in 10 of 105 answers, in both directions. After months of rewriting, it had absorbed the verdicts of some of the articles it was summarizing, and it repeated them in its own voice. It also started naming private people and labelling outlets, which no memory was asked to do.
None of this needs bad intent. Feed a rewriting memory enough opinionated text and some of the opinion becomes the memory.
3. A brief fixes the present, not the past
Telling the summary what to track helped a lot. On “what is everyone’s position right now”, the briefed summary (38%) was level with the ledger (41%), and on fresh, fast-moving news it often wrote the better answer. Here’s its answer on 25 April, the day one faculty’s student assembly voted to demand early elections:
“On 25 April the 17th plenum of FTN Novi Sad students in blockade voted to demand extraordinary parliamentary elections and to compile a list, and pro-government outlets cast this as proof the students’ aims were political all along. Other plenums and the wider movement have not been reported adopting this, so it is unclear whether it is a collective student position.”
That’s a good answer: careful about who decided what. Where the brief didn’t help is history. Asked whether anyone had changed their position over the months, the briefed summary found 13% of the changes in the answer key. Earlier positions are exactly what a rewrite throws away first.
4. The ledger notices changes, and invents a few of its own
The ledger found two to three times as many real changes of position. The sound device shows why. On 20 March, the day after the Interior Ministry’s account changed, the briefed summary still told the story well: “The ministry’s stance shifted from an initial denial that it had such a device, to saying it was in storage, to admitting ownership of the LRAD models.” Four days and two rewrites later, the same memory said:
Summary with a brief, 24 March: “A daily reported that police first denied owning LRADs and then said they were in storage, yet it found them mounted on police vehicles; no formal change of stance by any actor is recorded in the memory.”
The change had dissolved into “a daily reported”. The ledger, which never rewrites, still had it:
Ledger, 24 March: “The only stance change reported is the Interior Ministry, which moved from a partial position on 16 March to affirming on 18 March.”
It wasn’t perfect either: in the same answer the ledger still listed the President and the Speaker as denying that the state has such a device, although both had said by then that it does. A ledger is only as current as its last correctly read article.
But the ledger also reports changes that never happened. Asked on 8 February whether anyone had changed their position on the students’ demands, it listed eight changes, including the Speaker “moving from affirming to denying” that the demands were met. She never did: she had said they were met, and in late January that they were practically impossible to fulfill, but never that they were unmet. One misread article recorded her stance wrong, and the ledger faithfully showed a reversal. Over the whole test the ledger reported 604 changes, and for 284 of them the answer key has no change.
The fix is the same one a careful journalist uses: don’t believe a change until a second source confirms it. With a rule that a change only counts when two independent outlets report the new position, the ledger caught 41% of the real changes, reported a third fewer changes overall, and lost nothing on current positions. In my view this is the more useful lesson of the whole exercise. A rolling summary fails quietly, by forgetting. A ledger fails loudly, by flipping, and every flip points to the article that caused it, so you can check it.
5. The model alone knew nothing, and said so
I expected the model with no memory to “know how the story ends” and leak later positions into earlier dates. It didn’t. Asked what each actor’s position was on the documentation as of 20 March, it said:
“I have no memory provided on this proposition, and I cannot reliably recall any actor’s stated position on whether all documentation on the reconstruction of the Novi Sad railway station has been published as of 20 March 2025. I am therefore not reporting any attributed stances or stance changes.”
For a regional story at this level of detail, the model’s own knowledge wasn’t the risk. The memory was.
6. Search knows this week, not the season
Search (looking things up fresh for every question) was good at what had just happened. On the sound device, it described the Interior Minister’s change of account in detail, because the articles about it were the newest and most relevant. But across the whole six months it got only 16% of the current positions right, because no single search brings back everyone who said something since December. Search answers questions; it doesn’t keep track.
7. The ledger was also the cheap option
Six months of ledger cost $3.69. A cheap model (Claude Haiku 5.5) throws out excerpts with no statement in them, a stronger one (Claude Sonnet 5.5) writes the records, and both run through Anthropic’s batch service at half price, because every excerpt can be processed on its own. A rolling summary can’t be done that way: each rewrite needs the previous one, so it runs one step at a time, at full price, rewriting the whole text every time. Each summary cost about $7.40. The whole experiment, checking included, cost $25.40.
So, how many versions of the truth can an agent hold?
As many as you want, as long as it writes them down instead of rewriting them. The rolling summary didn’t break on any single contradiction. It broke on the accumulation: every rewrite kept the present and dropped a little of the past, until the disagreement it was supposed to track was mostly gone. That’s the point this whole series has been making: what decides whether an agent stays reliable is the knowledge layer around the model, not the model itself.
What this is, and what it isn’t
It’s a demo: one story, one language, one run of each memory, an answer key of 137 statements that is neither complete nor double-checked by a second person. The summaries were allowed to run longer than the ledger (about 700 words against a 600-word cap), which favors them, and on the busiest topic (the sound on 15 March) the briefed summary actually kept a few more statements than the capped ledger. It doesn’t say who was right about anything, and it doesn’t compare news outlets.
Run it yourself
The repository has the code, the answer key with a link to every source, the ledger, all the scores, and a viewer where you pick a topic and a date and see every actor’s position as it was known then, with the earlier ones underneath. It contains no article text: to rerun everything you fetch the articles from the outlets with the dataset’s own polite fetcher, then run five commands, each of which prints its cost before spending anything.
https://github.com/RatkoNikolic/srb-news-stance-ledger
Production checklist
Write statements down as records, then generate the prose. Never let a model rewrite its own memory in place.
Keep one record per speaker. Two sources disagreeing is two records, not one “current” value.
Confirm changes before you believe them. Require a second source before a record overturns an earlier one, and log every flip with the source that caused it.
Check what the memory still holds, on a schedule. Asking today’s question tells you nothing about what was lost from last quarter.
Test history separately from the present. A good brief fixes the present and not the past.
Batch the writing. Per-item extraction is parallel and half price; sequential rewrites are neither.
What to read next
The knowledge layer series:
Article 1 — Your agent’s problem isn’t the model, it’s the knowledge layer
Article 2 — Naive RAG: the baseline you’ll always benchmark against
Article 3 — Advanced RAG: what you actually run in production
Article 4 — GraphRAG and friends: when entities and relationships beat similarity
Article 5 — Agentic RAG: when retrieval becomes a decision, not a pipeline
Article 6 — Vector vs graph vs episodic: a tour of agent memory systems
Article 7 — Context engineering: the discipline that replaces prompt engineering
Article 8 — Knowledge editing and why it isn’t the answer for enterprise updates
Article 9 — Evaluation: how do you know your agent setup is working?
How many versions of the truth can an AI agent hold before it loses the plot? (BONUS)
Article 10 — Choosing your knowledge layer: a practical enterprise selection guide (coming soon)
External
Zep: A Temporal Knowledge Graph Architecture for Agent Memory (Rasmussen et al., January 2025, arXiv:2501.13956).
Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (Chhikara et al., April 2025, arXiv:2504.19413).
LLM as a Broken Telephone: Iterative Generation Distorts Information (Mohamed et al., ACL 2025).
Agentic Context Engineering (Zhang et al., ICLR 2026, arXiv:2510.04618).