Agent Memory — What to Remember and What to Forget

We've redesigned our DevOps agent's memory four times in the last nine months. Each redesign was triggered by the same one thing — we gave it more memory and behavior got worse.

This post is the conclusion of those nine months — agent memory isn't about what to store. It's about what to forget.

Memory = accumulating context

For an LLM agent to do multi-turn work, results from previous turns must enter the next turn's input. That accumulation is memory. The naive form is just append the entire conversation history.

It breaks at three points:

• Context window limits. Even 200K-token models tap out around 100 turns.
• Long-context accuracy decay. "Lost in the middle" — info in the middle of context gets ignored.
• Cost. Sending full history every turn makes token cost grow quadratically with turn count.

Four kinds of memory

The fix is to separate kinds of memory.

1. Working memory — current task context

The direct material of the current problem. User request, last action, last result. Short and volatile.

Storage: the recent portion of the context window. New turns push old ones out.

2. Episodic memory — past task outcomes

Past similar problems solved. "We've seen incidents like this before."

Storage: external vector store. Embedding match retrieves past episodes similar to the current task.

3. Semantic memory — domain knowledge

Org runbooks, policies, architecture docs. Changes slowly.

Storage: RAG (vector retrieval). Each turn fetches only relevant pieces.

4. Procedural memory — learned patterns

"This kind of task is solved through these steps" — generalized procedures. Usually baked into the model or fine-tuned.

Storage: examples in the system prompt or a separate fine-tune.

Four forgetting policies

The crux is how each memory forgets.

Working — time-based

Simplest. Working memory older than N turns is forgotten. Important facts get promoted to episodic.

Promotion criteria: decision changes, new facts, user corrections.

Episodic — similarity + frequency

Past episodes accumulating too many makes retrieval blurry. Episodes older than 6 months and infrequently accessed get compressed.

Compression = original episode → summary. The summary keeps the vector but drops the full text. Searchable, but lighter.

Semantic — explicit refresh

When domain documents change, the affected portion of the vector store is re-indexed. Not automated — humans decide which changes matter.

This is the most-frequently-broken layer. Many automate it, but judging which document changes are meaningful is hard for the model itself to do.

Procedural — fixed or fine-tune

Almost never forgotten. Updates happen on a fine-tuning cycle — quarterly or so.

Four traps we hit

1. Choosing when to compress working memory

How many turns is N? We started at 50 turns → reduced to 10 → settled at 15. 50 made context too long, accuracy fell. 10 dropped the immediately preceding task context. 15 was the balance.

2. Similarity threshold for episodic

How similar must an episode be to retrieve? We started at 0.7 cosine similarity → raised to 0.85. 0.7 pulled in too much loosely related stuff. 0.9 missed genuinely similar episodes.

3. Semantic staleness

Org policy docs sat in the vector store as 2-year-old versions. The agent acted on outdated policy. We added a quarterly explicit review.

4. Conflicts across the four

Hardest. Semantic says policy X; an episodic case from the past did Y. The model unstably picks one.

Fix: explicit priority — semantic > episodic > working. After a policy change, invalidating affected past episodes is explicit.

Nine months in one line

It's not give it more memory to make it smarter. It's give it less to make it more accurate. Half of memory design is the forgetting policy.

Avoiding the redesign

If your LLM agent gets less accurate as turns grow longer, the problem is usually not the model but a missing forgetting policy. When the four memory types (working / episodic / semantic / procedural) and their four forgetting policies aren't defined together in one place, a redesign awaits in nine months, like ours did. Defining them up-front is much cheaper.