Working, Episodic, Semantic, Procedural: The Four Agent Memories

Most "agent memory" is one undifferentiated blob: a vector store you dump everything into and hope recall sorts out. That works until it doesn't — until the transcript of this turn is competing for retrieval against a fact the contact told you six months ago, and the agent's own persona is somehow in the same bucket as a phone number.

The fix isn't a better embedding model. It's a taxonomy. CoALA — Cognitive Architectures for Language Agents — names four distinct memory kinds, each with a different lifetime, a different write path, and a different consumer. Matrix implements all four, and keeping them separate is what stops recall from turning to mush.

This post maps each CoALA memory kind to the exact mechanism behind it. If you're building an agent runtime, this is the part worth stealing.

The four kinds at a glance

The cardinal rule, and the one most home-grown systems get wrong: Session.summary is the digest of this session; Memory rows are what survives across sessions. Conflate them and either your recall fills with stale chatter or your durable facts vanish when the call ends.

WORKING — assembled fresh, every turn

Working memory is CoALA's "active and readily available information as symbolic variables for the current decision cycle." It is not stored. It is assembled, used, and discarded — every single turn.

In Matrix that's the WorkingMemory object (org.au.runtime.WorkingMemory). Both interactive and autonomous agents build it the same way, through WorkingMemoryAssembler.assemble(...), which reads long-term memory once into an immutable MemorySnapshot (org.au.memory.MemorySnapshot). Reading once matters: it's the difference between one Neo4j round-trip per turn and an N+1 storm.

What goes into the snapshot for a single cycle:

• the persona (the agent's procedural memory, more on that below)
• the objective — the per-call goal injected by a campaign, or the chat's standing intent
• recalled long-term memory — the SEMANTIC facts that survived prior sessions, vector-recalled for this contact
• the typed action space — every tool the agent can call this turn, classified REASONING · RETRIEVAL · LEARNING · GROUNDING
• progress so far — the running observations block

AgentPromptComposer.compose(WorkingMemory) turns that into the system prompt. The action-space and "Progress so far" blocks are gated — they render to "" unless populated — so the voice and chat prompts only carry weight they actually need.

The payoff of one assembler is parity. Text chat, telephony voice, browser-direct voice, and autonomous background tasks all PERCEIVE through the same path, so an agent behaves identically no matter how a contact reaches it. Working memory is the hub; everything else is what flows through it.

EPISODIC — what happened, step by step

Episodic memory is the log of experience: this turn, this step, this call. It's scoped to one interaction and it's where "what just happened" lives before any of it becomes a durable fact.

Matrix writes episodic memory in three shapes, depending on the channel:

Interactive: the post-call digest

When a voice or chat session ends, MemoryExtractorService runs a fire-and-forget LLM pass over the transcript and folds a digest onto Session.summary — not into a separate Memory row. This is deliberate. The digest is about this session; it belongs to the session.

MemoryContextRenderer later surfaces these as "recent conversations" by scanning recent Session rows for the same (agent, userId) that carry a non-blank summary. So next time the contact calls, the agent can reference last time — without that recap leaking into the cross-session fact pool.

Autonomous: the step log

An autonomous agent (Agent.mode = AUTONOMOUS) runs a PERCEIVE → DECIDE → ACT → LEARN loop with no human in it. Each pass through the loop appends to TaskRun.stepsJson — one entry per propose → evaluate → select → act → observe cycle:

loop (step ≤ maxSteps):
  wm   = assemble(AUTONOMOUS, goal, retrieved-memory, action-space, observations)  # PERCEIVE
  d    = autonomousDriver.decide(wm)                                                # DECIDE
  if d.finish: write EPISODIC result; return completed
  res  = toolDispatcher.invoke(toolCtx, callbacks, d.tool, d.args)                  # ACT
  observations += summarize(d, res)                                                 # OBSERVE
  write EPISODIC step                                                               # LEARN
return budget-exhausted

Alongside stepsJson, the loop writes a per-step Memory row tagged EPISODIC, plus one for the final result. That gives you both a structured replay of the run and recallable traces of individual steps. The run ends COMPLETED when the objective is met or FAILED when the step budget runs out.

The unifying idea: episodic memory is bounded by the interaction. A digest covers one session; a step log covers one task run. Neither is meant to outlive its scope unless something promotes it — which is exactly what semantic memory is for.

SEMANTIC — facts that outlive the conversation

Semantic memory is the agent's durable knowledge about the world and the people in it: the contact's date of birth, their city, the note that they prefer mornings. It crosses session boundaries. In Matrix it's the Memory row — and it's filled by two paths.

The extractor's new-note path. The same post-call pass that writes the Session.summary digest also pulls out durable facts worth keeping and writes them as SEMANTIC Memory rows. The digest stays on the session; the facts graduate to the cross-session pool.

The LEARNING tools. Every agent gets five memory tools for free — update_contact_profile, set_contact_birth_place, set_contact_current_location, add_contact_note, lookup_contact_details — plus the generic update_lead / update_record writers. When the agent calls one of these mid-conversation, it writes a SEMANTIC Memory row directly, in the moment, instead of waiting for the post-call sweep.

Both paths embed on write (768d) and store the vector on Neo4j's native HNSW index, so recall is a real vector search scoped to (agentId, userId) — one pool per contact, shared across chat and phone because Session.userId joins them. The agent remembers you whether you call or type.

Two refinements keep this pool from rotting, both on the shared LEARN path:

• Reconcile-on-write. Before writing a new SEMANTIC note, Matrix embeds it and compares to the nearest existing memory in scope. Cosine at or above 0.95 → UPDATE (supersede the old row, keep it for history). Below 0.80 → ADD. In between → an optional LLM arbiter decides. Near-duplicates stop accumulating.
• Temporal validity. Rows carry validFrom, supersededAt, and a supersedes back-link. Recall excludes superseded rows, so the latest fact wins while history is retained.

We cover both in depth in Reconcile-on-Write. The line to hold onto here: Memory rows are what survives across sessions; Session.summary is not.

PROCEDURAL — the agent's own behaviour

CoALA's fourth memory is the sleeper: procedural memory is how the agent acts — its persona, its skills, its rules. In Matrix this is the agent's identity, and it's already stored where everything else is: as EntityType / EntityNode rows in Neo4j. The persona, the attached skills' prompt blocks, the memory-field requirements — all data.

Two properties make this real procedural memory rather than a static config file:

1. It's recomposed per turn. The prompt is rebuilt from the live entity on every cycle through AgentPromptComposer. There's no baked-in prompt string to redeploy.
2. The agent cannot edit it. This is the safety posture, and it's structural, not policy. There is no tool that writes to an Agent, Skill, or Tool entity. The most an agent can do — and only when explicitly enabled — is propose a change via the propose_procedural_edit LEARNING action, which writes a PROPOSED record that a human reviews and approves before it takes effect.

That self-improvement path ships dark: it's double-gated behind Agent.selfImproveEnabled per agent and a platform kill-switch, both default off. When it is on, an approved edit takes effect on the agent's next turn — because procedural memory recomposes per turn from the live entity, no redeploy needed. The full design, including why "the agent never mutates its own code; it only ever proposes" is enforced by construction, is in Self-Improving Agents That Can't Go Rogue.

Why the boundaries earn their keep

It's tempting to collapse these into one store. Don't. Each boundary does work:

• WORKING vs. the rest keeps the per-turn assembly cheap (read long-term memory once) and the prompt the same across every channel.
• EPISODIC vs. SEMANTIC is the rot guard. Session digests would crowd recall if they lived in the durable pool; durable facts would evaporate if they lived on the session. Keeping Session.summary and Memory rows apart is what lets recall stay sharp call after call.
• PROCEDURAL apart from everything is the safety boundary. Because the agent's behaviour is data the agent cannot write, even a fully compromised agent can't rewrite its own persona — it can only propose.

Takeaway

Four memory kinds, four mechanisms, four lifetimes:

• WORKING → the WorkingMemory object, assembled fresh and read once per turn.
• EPISODIC → Session.summary (interactive) and TaskRun.stepsJson + per-step Memory rows (autonomous).
• SEMANTIC → durable Memory rows, vector-recalled per contact, kept clean by reconcile-on-write and temporal validity.
• PROCEDURAL → the agent's persona and skills, recomposed per turn and editable only through the gated propose→approve flow.

If you remember one thing, remember the split: a session digest is not a durable fact. The taxonomy is the architecture.

For how this same PERCEIVE → DECIDE → ACT → LEARN cycle drives both interactive and autonomous agents from one runtime, read CoALA in Production.

Build an agent that remembers — across chat and phone, with no memory infra to wire up. Create a workspace, spin up an agent in /admin/agents, and the four memory kinds come attached. The runtime is documented in docs/COGNITIVE_CORE.md if you want to read the source first.