AI Agent Memory Systems: How We Keep 13 Agents in Context Across Sessions
Why Do AI Agents "Forget"? Ever had this experience: you spend half an hour with an AI building out a plan, it's all mapped out perfectly, then you open a ne…
Why Do AI Agents "Forget"?
Ever had this experience: you spend half an hour with an AI building out a plan, it's all mapped out perfectly, then you open a new conversation the next day and it remembers nothing. That's not a bug — it's a fundamental architectural design of LLMs. Context windows are finite, and they clear when the session ends.
But at SFD Lab, our 13 Agents collaborate on dozens of tasks every day, spanning sessions, channels, and days. If every session starts from zero, efficiency takes a massive hit.
Our Memory Architecture: Three Layers
After nearly 20 days of production use, we worked out a three-tier memory model.
Layer 1: Working Memory (Session Context)
This is the LLM's Context Window. Fast, precise, but temporary. Used for instruction passing and return values during active task execution. Session ends, it's gone.
Layer 2: Daily Memory (Daily Notes)
Every Agent has a memory/YYYY-MM-DD.md file. After completing a task, it immediately writes: what was done, what went wrong, what downstream steps were triggered. Start of next day, read yesterday's file and recover context in five minutes.
Layer 3: Long-term Memory (MEMORY.md)
The distilled "brain." Not daily logs, but cognition worth keeping long-term: architecture decisions, lessons from failures, team standards. Reviewed weekly, outdated items deleted, duplicates merged.
Why Three Layers?
Single-layer solutions all have failure modes. Pure context window: cost explosion at scale. Pure file system: retrieval inefficiency. Pure vector database: operational complexity too high. Three layers covers the full spectrum: real-time (context) → short-term (daily) → long-term (MEMORY).
Real Lessons from the Field
Lesson 1: The trigger for writing is more important than what you write. Early on, Agents would forget to write memory after completing tasks. We eventually added memory write requirements directly to the task specification template — automatic triggering, not relying on the Agent's initiative.
Lesson 2: Memory needs structure to be retrievable. Unstructured daily notes are hard to search. We standardized a format: Project tag, conclusion, file changes, lessons, hashtags. Hit rate in memory_search went from ~30% to ~75%.
Lesson 3: Memory maintenance is a task in itself. Without periodic review, MEMORY.md grows without bound and retrieval quality degrades. We now dedicate one heartbeat per week specifically to memory maintenance — reading recent daily logs and distilling what's worth keeping long-term.
Current Token Cost
Before the three-tier system: each session started with 8-15k tokens of context (re-explaining background). After: daily notes pre-load is 1-3k tokens, MEMORY.md loads are 2-4k tokens, total overhead roughly halved, and task completion quality improved significantly because context is actually relevant.