AI Agent Memory Architecture: Build 3 Memory Layers

An ai agent memory architecture is the storage and retrieval system that decides what an agent remembers, what it forgets, and what it can safely bring back into the next model request. The architecture is not a single vector database. A useful production design separates at least three layers: working memory for the current task, vector memory for episodic recall, and summary memory for durable facts and preferences.

This article is about how to design, implement, retrieve from, expire, and evaluate those layers. A separate existing article explains why agents lose coherence over long sessions; see why most AI agents fail after 30 minutes and how memory fixes it. Here, the focus moves from motivation to architecture.

The central rule is simple: do not append everything to the context window. Keep the current working state small, persist useful episodic events as retrievable vectors, and compress durable meaning into summary records. Then make retrieval, decay, permissions, and evaluation explicit parts of the system.

Why an AI Agent Memory Architecture Needs Three Layers

Memory is often described as one feature, but production agents need several lifetimes of data:

These layers answer different questions. Working memory answers, "What is happening now?" Vector memory answers, "What similar situation has this user or agent seen before?" Summary memory answers, "What should survive even when the raw conversation is no longer useful?"

If all three are mixed into one message history, three predictable problems appear. First, old tool output crowds out the current instruction. Second, sensitive or expired information remains available forever. Third, the model has no way to distinguish a temporary observation from a durable user preference.

Separate Model Context from Memory Storage

The model context is a request-time budget. Memory storage is a durable system of record. They should never share the same lifecycle.

+-------------------+
             |   User request    |
             +---------+---------+
                       |
                       v
              +--------+--------+
              | Memory selector |
              +---+-------+---+-+
                  |       |   |
     +------------+   +---+   +------------+
     |                |                |
     v                v                v
Working memory   Vector memory    Summary memory
current task     similar events   durable facts
     |                |                |
     +--------+-------+--------+-------+
              |                |
              v                |
        Context budget         |
              |                |
              v                v
           Model adapter <-----+

The selector is the important part. It receives the current task, retrieves candidates from long-term stores, ranks them, enforces permissions and expiry, and assembles a context that fits the request budget. Only then does the model adapter send the request.

Define a Record Contract Early

Every memory record should have enough metadata to be retrieved, audited, expired, and deleted. A practical record includes:

memory_id
owner_or_tenant_id
agent_id
session_or_thread_id
memory_type: working | episodic | summary
content
summary
embedding
created_at
last_accessed_at
expires_at
importance
confidence
source_turn_or_event_id
access_scope
pii_flags
consent_state
version

The exact storage can be Postgres, Redis, a vector database, or a combination. The contract matters more than the engine. Without owner, source, time, scope, and consent fields, the memory becomes difficult to govern as the product grows.

Implement Working Memory Without Turning Context into a Database

Working memory is the agent's active scratchpad. It should contain the current user request, pinned constraints, a bounded number of recent turns, unresolved tool calls, and temporary variables needed by the workflow.

Keep working memory small and deterministic. A good default is to treat it as a structure with explicit slots, not as an ever-growing list:

from dataclasses import dataclass, field
from datetime import datetime


@dataclass
class WorkingMemory:
    task: str
    pinned_constraints: list[str] = field(default_factory=list)
    recent_turns: list[dict] = field(default_factory=list)
    tool_results: dict[str, dict] = field(default_factory=dict)
    status: str = "active"


    def add_turn(self, role: str, content: str, *, max_turns: int = 8) -> None:
        self.recent_turns.append(
            {"role": role, "content": content, "at": datetime.utcnow().isoformat()}
        )
        self.recent_turns = self.recent_turns[-max_turns:]


    def attach_tool_result(self, call_id: str, result: dict) -> None:
        self.tool_results[call_id] = {
            "result": result,
            "state": "unread",
        }

The example uses a short-term Python object because working memory often belongs to the request or task run. If your process is multi-worker, put this state in Redis or a database with a task-run ID and a TTL. The important part is not the storage product; it is the bounded, typed contract.

Compress Tool Output Before It Fills the Context

A single API response can contain thousands of tokens. If the agent only needs a decision, store the structured facts and return a compact projection to the model.

def project_ticket_for_model(ticket: dict) -> dict:
    return {
        "ticket_id": ticket["id"],
        "status": ticket["status"],
        "priority": ticket["priority"],
        "customer_tier": ticket["customer_tier"],
        "open_questions": ticket["open_questions"],
    }

The full result can remain in application storage. The model only needs fields that affect the next action. This preserves working-memory budget and reduces the chance that a stale stack trace or customer PII leaks into later turns.

Build Vector Memory for Episodic Recall

Vector memory stores what happened: past conversations, user-uploaded documents, task observations, resolved incidents, and prior tool outputs. The embedding lets the agent retrieve semantically similar situations, while metadata keeps retrieval scoped to the right user, tenant, agent, and time window.

A basic ingestion path looks like this:

event or message
      |
      v
normalize and redact
      |
      v
chunk by semantic boundary
      |
      v
create embedding
      |
      v
write content + embedding + metadata

Do not embed everything unchanged. Remove secrets and fields the user has not consented to store, split long documents at semantic boundaries, and give every chunk a stable parent ID so it can be traced back to the source event.

Embedding Example with an OpenAI-Compatible Endpoint

The following example is intentionally small. It uses the OpenAI SDK against an OpenAI-compatible embeddings endpoint and a local list instead of assuming a particular vector database. Replace the list with your production index and use the exact model ID available to your account.

import os

from openai import OpenAI


client = OpenAI(
    api_key=os.environ["OURTOKEN_API_KEY"],
    base_url="https://api.ourtoken.ai/v1",
)


def embed_texts(texts: list[str], model_id: str) -> list[list[float]]:
    response = client.embeddings.create(
        model=model_id,
        input=texts,
    )
    return [item.embedding for item in response.data]


class LocalVectorMemory:
    def __init__(self, model_id: str):
        self.model_id = model_id
        self.records = []


    def add(self, memory_id: str, content: str, metadata: dict) -> None:
        embedding = embed_texts([content], self.model_id)[0]
        self.records.append(
            {
                "memory_id": memory_id,
                "content": content,
                "embedding": embedding,
                "metadata": metadata,
            }
        )


    def search(self, query: str, *, top_k: int = 5) -> list[dict]:
        query_vector = embed_texts([query], self.model_id)[0]

        def dot(a: list[float], b: list[float]) -> float:
            return sum(x * y for x, y in zip(a, b))

        ranked = sorted(
            self.records,
            key=lambda record: dot(query_vector, record["embedding"]),
            reverse=True,
        )
        return ranked[:top_k]

For production, use a vector index that supports approximate nearest-neighbor search, metadata filtering, tenant isolation, and deletion. The local class is useful for tests because it makes the retrieval contract visible.

Chunking and Metadata Are Part of Memory Design

Chunk size affects both recall quality and embedding cost. Very large chunks reduce index precision and waste context. Very small chunks lose local meaning and return fragments that require expensive reranking.

Start with a measurable policy rather than a magic number:

Later, evaluate these choices with the same discipline you would use for a RAG system. The RAG evaluation metrics guide is directly applicable because agent memory retrieval is a retrieval problem before it is a generation problem.

Build Summary Memory That Survives Token Budgets

Summary memory is not just a shorter transcript. It is a set of durable, reviewable records that capture goals, preferences, decisions, constraints, and procedures. A raw chat log answers "what was said"; a good summary record answers "what matters for future work."

Use structured records instead of one giant narrative summary:

from datetime import datetime


def create_summary_record(
    owner_id: str,
    subject: str,
    statement: str,
    source_memory_ids: list[str],
    confidence: float,
) -> dict:
    return {
        "memory_type": "summary",
        "owner_or_tenant_id": owner_id,
        "subject": subject,
        "content": statement,
        "source_memory_ids": source_memory_ids,
        "confidence": confidence,
        "created_at": datetime.utcnow().isoformat(),
        "version": 1,
    }

Examples:

{
  "subject": "deployment_preferences",
  "content": "The team prefers canary releases with a 5 percent first stage and a one-hour observation window.",
  "source_memory_ids": ["event_812", "event_904"],
  "confidence": 0.86
}

{
  "subject": "support_context",
  "content": "The customer uses SSO with Okta and has two production workspaces: EU and US.",
  "source_memory_ids": ["event_109", "event_233"],
  "confidence": 0.94
}

Keep summaries independent when possible. Do not merge a user preference, a temporary project constraint, and an account fact into one paragraph. Separate records can be updated, revoked, expired, and audited independently.

Use a Deliberate Write Policy

Not every conversation should update summary memory. A safer policy has three stages:

1. Candidate: the model proposes a durable fact with source references.
2. Validate: a rule engine checks schema, PII, scope, and contradiction with existing records.
3. Commit: the record becomes retrievable, or enters a review queue for sensitive categories.

The prompt should ask for structured output rather than free-form memory updates:

Extract only durable information relevant to future tasks.
Return a JSON object with these fields:
- subject: short stable category
- statement: one concise sentence
- confidence: 0.0 to 1.0
- memory_type: preference | fact | decision | procedure
- source_turn_ids: exact IDs from the current task

Do not include secrets, temporary goals, payment credentials, or information the user asked not to remember.

Then validate the output against a JSON Schema before writing it. If the statement conflicts with an existing summary, create a new version and preserve the old source records until your retention policy says otherwise.

Retrieve with Recency, Similarity, and Importance Together

Similarity search alone is not memory management. A six-month-old message can be semantically close to the current question but no longer relevant. A recent event can be close but temporary. A durable preference can be dissimilar in wording but essential.

Rank candidates with explicit signals:

retrieval_score =
  semantic_similarity
  + recency_bonus
  + importance_bonus
  + task_match_bonus
  - stale_penalty
  - permission_penalty
  - duplicate_penalty

The weights are product decisions. Start with simple, observable values and tune them with evaluation data. A retrieval candidate that fails permission filtering should never be sent to the model, even if its semantic score is high.

Assemble Context Under a Budget

After candidates are ranked, allocate the remaining request budget in a fixed order:

1. system and safety instructions
2. pinned working-memory constraints
3. selected summary records
4. selected episodic memories
5. recent conversation turns
6. current tool output

Keep a reserve for the model response. If candidates exceed the budget, include the highest-ranked items and provide a short provenance note:

Relevant prior context:
- [pref-12] The user prefers concise answers with code examples.
- [event-88] On June 2, the staging deployment failed because of a missing migration.
- [doc-4, section 3] The current service supports two database regions.

Provenance helps the model avoid treating retrieved context as universal truth. It also helps a reviewer understand why the agent made a decision.

Prevent Retrieval Feedback Loops

Agent output can become memory. If the system writes every assistant response back into the index and retrieves it later, small mistakes compound.

Use these safeguards:

• Do not write model output as episodic memory unless it was verified by a tool, user, or rule.
• Mark unverified model statements with a lower confidence and a shorter TTL.
• Keep source and generated content in separate indexes or namespaces.
• Add a duplicate check before writing similar records.
• Allow an operator to quarantine an agent-generated memory.

Handle Forgetting, Expiry, and User Control

Forgetting is a first-class capability. Without it, memory becomes a liability and retrieval quality slowly gets worse.

A forget operation should not merely hide the row. If the record contains content and an embedding, remove or cryptographically invalidate both. Propagate deletion to derived records where legally or operationally required, and preserve only the minimum audit metadata allowed by your policy.

Implement consent as a state machine, not a checkbox buried in settings:

not_collected -> requested -> granted | denied
granted -> active | paused
active -> revoked
revoked -> delete_requested -> deleted

When consent is paused, retrieval must skip records even if they remain in storage. When consent is revoked, the delete workflow should run asynchronously but produce a visible status to the user.

Test Memory Quality Before Production

A memory system changes model behavior silently, so it needs its own regression suite. Evaluate retrieval and end-to-end behavior separately.

Build a small golden set from real task patterns: repeated questions, contradiction cases, expired context, tenant boundaries, and sensitive data. Run it before every schema or model change.

For online quality, log non-sensitive telemetry such as:

task_id
agent_id
owner_or_tenant_id
query
selected_memory_ids
memory_scores
context_tokens
summary_tokens
working_memory_tokens
latency_ms
model_id
answer_rating
contradiction_flag

Do not log secrets or raw sensitive content. Keep enough metadata to explain why a memory was selected and to reproduce a failure without reproducing private data.

Route Memory Workloads Through One Model Boundary

Memory operations use models for different jobs: extracting candidate summaries, answering with retrieved context, reranking candidates, or explaining why an agent made a decision. Put these model calls behind the same adapter boundary as the rest of the agent.

For example, a team might use GPT-5.6 Terra for difficult extraction or reasoning, GLM 5.2 for summary generation where it meets quality requirements, and DeepSeek V4 Pro for selected analytical workloads. These are candidate routes, not universal recommendations. Verify exact model IDs and feature support in the live OurToken model catalog.

The LLM model routing guide explains how to route simple and complex work deliberately. For memory, routing has an additional rule: the memory selector should work even when the model call fails. Returning a safe, reduced-context answer is better than fabricating a memory.

If the agent uses tools to read tickets, search documents, or update records, keep those capabilities separate from memory storage. The OpenAI-compatible tool calling guide covers the transport pattern; the memory architecture should record only the observations that your write policy approves.

Production Checklist

Before enabling long-term agent memory, verify:

[ ] Working memory is bounded and cleared on task completion.
[ ] Episodic records have owner, tenant, session, source, and expiry fields.
[ ] Embeddings are scoped and filtered before ranking.
[ ] Sensitive fields are redacted or encrypted before storage.
[ ] Summary records are structured, versioned, and traceable to source IDs.
[ ] Contradictions create a new version instead of silently overwriting history.
[ ] Retrieval combines similarity, recency, importance, permissions, and TTL.
[ ] Context assembly has a token budget and a response reserve.
[ ] User pause and revocation controls work end to end.
[ ] Deletion removes content and embeddings according to policy.
[ ] Retrieval precision, recall, permission, freshness, and answer quality are tested.
[ ] Model routes are approved, monitored, and safe to fall back.
[ ] Agent-generated memories are labeled and cannot silently become ground truth.

Conclusion

A practical AI agent memory architecture keeps three lifecycles separate. Working memory carries the current task and remains small. Vector memory preserves useful episodic context and retrieves it under metadata, permission, and freshness constraints. Summary memory distills durable meaning into structured, versioned records that can be updated and revoked.

The hard work is not embedding text. It is defining what deserves to be remembered, how it is retrieved, when it expires, and how you prove that the result improves agent behavior. Build the record contract, retrieval policy, and evaluation set before adding more memory features. Then route model calls through one explicit boundary so the system remains testable as models and providers change.

FAQ

What is an AI agent memory architecture?

It is the set of stores, schemas, retrieval rules, permissions, and expiration policies that determine what an agent remembers across turns or sessions. A robust design usually includes working, episodic vector, and summary memory.

Is vector memory enough for an agent?

No. Similarity search is useful for episodic recall, but it does not manage current task state, durable preferences, permissions, contradictions, or expiry. Vector memory works best as one layer inside a broader memory selector.

What is the difference between summary memory and conversation history?

Conversation history records what was said. Summary memory stores validated, durable meaning such as a user preference, project decision, account fact, or procedure. Summaries should be structured, sourced, versioned, and independently revocable.

How many turns should working memory keep?

Start with the smallest number that passes task tests, such as four to eight recent turns plus pinned constraints. The right value depends on task length and token budget; measure answer quality rather than copying another application's setting.

How should agent memory handle personal data?

Classify sensitive fields before storage, redact or encrypt them, scope every record to an owner or tenant, enforce consent during retrieval, and support deletion of both content and embeddings. Keep only the minimum audit metadata required by your policy.

How do I stop old memories from dominating responses?

Combine similarity with recency, importance, task match, permission, and stale penalties. Expire temporary records, supersede outdated summaries, and include provenance so the model can distinguish retrieved context from current instructions.

Should agent responses be stored in memory?

Only when they are verified by a user, tool, rule engine, or evaluation process. Otherwise label them as unverified, reduce their confidence, shorten their TTL, or keep them out of the retrieval index.

How do I evaluate memory quality?

Test retrieval recall, precision, permission boundaries, freshness, expiry, context-budget compliance, and final answer quality on a fixed evaluation set. Also monitor online signals such as contradiction flags, answer ratings, latency, and memory-selection traces.