Memory Is a Control Surface
The point of persistent memory is not to make the current loop feel smarter. It is to make the next loop cheaper, shorter, and less likely to repeat a known failure. In an overnight harness, context is a liability as much as an asset: every extra page competes with the live task, increases drift, and makes verification harder to interpret. Treat memory as part of the control plane, not as a transcript archive.
A practical layout starts with a lean agent entry file, such as `AGENTS.md` or `CLAUDE.md`, that contains only routing rules: where specs live, where run logs live, how decisions are recorded, which checks are mandatory, and what memory should be read for each class of work. The file should point, not explain. The agent should load a local index, choose specific concept files, then proceed. If the identity file becomes a handbook, it has already failed.
Use a predictable markdown bundle format for the durable layer. Require frontmatter at minimum with `type`, and add `title`, `tags`, `status`, `owner`, `related`, and `last_verified` where useful. Put indexes at the top level and inside each major folder so an agent can progressively disclose context: first the map, then the relevant node, then maybe one linked failure note. This gives you interoperability across Codex, Claude Code, Goose, local agents, and future wrappers without teaching each tool your private taxonomy.
Separate State by Job
Once memory is a control surface, the next design choice is separation. Do not store specs, gotchas, run logs, decisions, and retrospectives in the same pile. They have different read patterns and different half-lives. A spec constrains the target. A gotcha constrains behavior. A run log provides evidence. A decision records why an option was chosen. A retrospective changes the harness.
A usable repo-level structure is: `memory/specs/`, `memory/gotchas/`, `memory/decisions/`, `memory/runs/`, `memory/failures/`, `memory/retros/`, plus `memory/index.md`. Specs should be canonical and sparse, with stable IDs the harness can cite. Gotchas should be aggressively curated and small: one failure mode, one prevention rule, one verification signal. Run logs should be append-only and machine-readable enough to parse, often markdown with YAML blocks or JSONL alongside it. Retrospectives should not be loaded by default; they are inputs to memory maintenance.
The failure mode is obvious: teams convert every session into documentation, then wonder why agents ignore it. The fix is promotion. Raw session output goes into `runs/`. A retrospective agent or review pass extracts only durable lessons into `gotchas/` or updates the spec. Most logs should never enter primary context. They exist so the next loop can query them, compare against prior attempts, and produce evidence when something regresses.
Make Retrieval Boring
After separation, retrieval should become deterministic enough that the model cannot improvise the memory strategy. Give the harness a small memory CLI or script: list bundles, show indexes, read by concept ID, search tags, and fetch recent failures for a subsystem. For database-backed memory, keep the same interface. The agent should not care whether the backing store is markdown, SQLite, or a vector database. It should ask for `gotchas:billing-migrations` or `failures:last-10:playwright`, not scan the world.
Vector search is useful for long-term recall, but it should not be the only path. Embeddings are good at finding adjacent language, not at enforcing authority. Route through structured metadata first, then use semantic search inside the narrowed set. For example: load `memory/index.md`, identify `type: gotcha` and `tags: [frontend, auth]`, then embed-search only those notes. This prevents the agent from pulling a stale brainstorming note over the current spec because the words happen to be similar.
Compression belongs at the boundary. Tool output, logs, RAG chunks, and previous failures can be summarized before they reach the main context, especially in repeated loops where token cost dominates. But compressed memory must preserve citations back to raw evidence: test command, commit SHA, log path, screenshot path, hash, video artifact, or CI run ID. A summary without provenance is just another model claim.
Close the Loop With Evidence
Memory helps only if the loop writes back the right things. A long-running harness should move through explicit states: research, plan, implement, verify, review, retrospective. Use fresh or compacted context between phases when possible. Let sub-agents do bounded analysis in parallel, such as security review, performance review, failing-test diagnosis, or codebase reconnaissance, but make them return findings rather than broad edits. The main loop owns integration and state transitions.
Verification should be code-gated, not prose-gated. Require proof of work for claims that matter: test logs with hashes, Playwright recordings, screenshot paths, benchmark outputs, lint results, migration dry runs, or reproduced failure commands. Store those artifacts in the run record. The reviewer step should emit pass, fail, or score against the definition of done. If the harness accepts “looks good” as completion, memory will faithfully preserve low-quality evidence and make future loops worse.
The writeback policy should be narrow. On success, record the final state, commands run, artifacts, cost, token counts if available, and any new constraints discovered. On failure, record the smallest reproducible failure, suspected cause, attempted fixes, and the next recommended probe. Then run a retrospective pass that decides whether to update a gotcha, a spec, a hook, a skill, or nothing. Treat repeated agent mistakes as harness bugs. If the same failure appears twice, add a deterministic check or a small memory note. If it appears three times, redesign the loop.
Budget Memory Like Infrastructure
The final constraint is cost. Advanced builders often overfit on model quality and underfit on memory economics. Track session JSONL or equivalent logs across agents: wall-clock time, input tokens, output tokens, cache reads, tool calls, failed commands, and retries. Compare the same workload across models and tools. Use stronger models for ambiguous research, planning, and taste-heavy review; route bounded implementation, log summarization, and checklist verification to cheaper models when the evidence gate is objective.
Memory also has maintenance cost. Identity files rot slowly. Skills and workflow instructions rot faster. Materialized agents rot fastest because model behavior and tool conventions change underneath them. Keep a `memory/rot.md` or database table with review cadence by layer. Schedule pruning. Delete low-signal skills. Collapse verbose docs into gotchas. Move stale decisions to archived status instead of letting them compete with current constraints.
The target architecture is not a giant second brain. It is a small active context, a navigable durable store, and a loop that writes evidence back into the right shelf. That is what lets an agent run all night without becoming unmoored: it starts from the minimum relevant state, proves what it did, preserves only lessons that should alter future behavior, and leaves the next loop with less to guess.
Key takeaways
- Keep `AGENTS.md` or `CLAUDE.md` as a pointer layer, not a documentation dump.
- Separate specs, gotchas, decisions, failures, run logs, and retrospectives because each has a different read pattern and half-life.
- Use structured metadata and indexes before vector search so retrieval is authoritative before it is semantic.
- Require proof-of-work artifacts in run memory: hashes, logs, screenshots, recordings, CI IDs, or benchmark outputs.
- Promote only durable lessons from logs into active memory; most session output should remain archive material.
- Track token, time, retry, and failure metrics so memory design and model routing can be optimized empirically.