Memory Engine

TL;DR

Personal memory as a first-class engine on the ai-gateway. The Cloudflare Worker is the control plane and the sole billing emitter — multi-tenancy, rate limits, ownership re-check, per-user usage events to payment-gateway. The Railway side runs 100% vanilla mem0ai/mem0 — no source patch, no extension files, no factory-dict swaps. Configuration only, plus one init.sql that seeds the pgvector extension and mem0's default User row. Worker→engine auth uses mem0's stock ADMIN_API_KEY path: X-API-Key: <MEM0_ADMIN_API_KEY> matched via secrets.compare_digest. Mem0 talks to Google's Gemini API directly via its native provider: "gemini" (uses the google-genai SDK, properly translates response_format → Gemini's response_mime_type). Vector store: native 3072-d gemini embeddings stored as vector(3072) with hnsw=False — exact sequential scan inside each user's partition, best retrieval quality, fast enough at v1 per-user volumes. Billing is by request count: the Worker emits one { operation, count: 1 } usage event per successful MCP tool call. Credit pricing per operation lives on payment-gateway, not in the Worker. MCP at /engines/memory/mcp matches the existing /engines/<id>/mcp convention.

Architecture

Components

ai-gateway changes (src/)

Memory engine (infra/mem0/) — 100% vanilla mem0

The engine is the vanilla mem0ai/mem0 Docker image, unmodified. No COPY over source. No Python extension files. No factory-dict swaps. Zero lines of our own Python. Two things make this possible:

1. Mem0 has a native provider: "gemini" for both LLM and embedder (uses Google's google-genai SDK). It translates response_format={"type": "json_object"} to Gemini-native response_mime_type: "application/json", so no subclass is needed to coerce JSON output.
2. hnsw=False uses mem0's stock pgvector schema (vector(<dim>)) with no index — exact sequential scan over the user's partition. Avoids pgvector's 2000-d HNSW cap, so no schema subclass is needed for native 3072-d embeddings. At v1 per-user volumes (hundreds to low thousands of memories) seq scan is sub-millisecond and gives better recall than HNSW would (exact vs. approximate).

MCP Tool Surface

All seven tools live at /engines/memory/mcp, mounted via enginesApp exactly like image-gen. The Worker translates each tool to direct calls against mem0's native REST endpoints.

What infer=False vs infer=True means in mem0

Mem0's add() API has two paths through it, controlled by the infer argument. The choice matters because it determines where the LLM call happens, which in turn determines who gets billed.

Concrete example — same input, two outcomes

Input: [{user: "I just adopted a Welsh Corgi named Otis. He loves fetch."}, {user: "He's 8 weeks."}]. Existing memory for the user: "User has a dog named Otis."

• With infer=False: two new rows stored verbatim. Existing "User has a dog named Otis" untouched. Total: 3 memories, partially redundant.
• With infer=True: one LLM call extracts → UPDATE old="User has a dog named Otis" → "User adopted Otis, a Welsh Corgi, 8 weeks old" + ADD "Otis enjoys playing fetch". Total: 2 memories, deduplicated and consolidated.

Stop-hook wiring (the ingest path)

curl -sS -X POST "$GATEWAY_URL/engines/memory/mcp" \
  -H "Authorization: Bearer $USER_TOKEN" \
  -H "Content-Type: application/json" \
  -d @- <<'EOF'
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "ingest",
    "arguments": {
      "messages": [
        { "role": "user",      "content": "..." },
        { "role": "assistant", "content": "..." }
      ],
      "metadata": { "session_id": "abc123" }
    }
  }
}
EOF

Validation Findings

Validated end-to-end on 2026-05-18 / 19 against a local docker-compose stack: pgvector/pgvector:pg16 + mem0ai==2.0.2 + mem0's native provider: "gemini" pointed at Google's Gemini API with gemini-3-flash (LLM) and gemini-embedding-2 (embeds). Full add/search/update/delete cycle confirmed; example infer=True extraction from a three-message turn: "User adopted a Welsh Corgi named Otis around May 19, 2026, who loves playing fetch."

Mem0 response shapes (captured verbatim)

// add(infer=False)
{
  "results": [
    { "id": "...", "memory": "User is allergic to peanuts.",
      "event": "ADD", "actor_id": null, "role": "user" }
  ]
}

// add(infer=True) — three-message turn, one extracted fact
{
  "results": [
    { "id": "...",
      "memory": "User adopted a Welsh Corgi named Otis around May 19, 2026, who loves playing fetch.",
      "event": "ADD" }
  ]
}

// search — note score, metadata round-trips, role on user-source memories
{
  "results": [
    { "id": "...", "memory": "...", "hash": "...",
      "metadata": { "pinned": true, "source_type": "explicit" },
      "score": 0.8834746264057188,
      "created_at": "...", "updated_at": "...",
      "user_id": "...", "role": "user" }
  ]
}

// update — message only, no row data
{ "message": "Memory updated successfully!" }

Multi-tenancy And Security

Memories are separated by user. Three enforcement layers in the Worker, plus a Worker-side ownership re-check that substitutes for same-DB-transaction enforcement (which is no longer possible without engine code).

1. Type-enforced scope derivation

   MemoryScope is a branded type whose only constructor reads JWT.sub post-verification. Mem0Client methods accept only MemoryScope. Passing a raw string is a compile-time error.

2. Request-body sanitization

   Any user_id-shaped field in MCP tool arguments is dropped before forwarding. Only the server-derived scope reaches the engine.

3. Admin API key gate at the engine

   Inbound calls must carry X-API-Key: <MEM0_ADMIN_API_KEY>, matched by mem0's stock verify_auth via secrets.compare_digest (constant-time). The key is a Wrangler secret on the Worker side and a Railway env var on the engine side — same shared-secret posture as USAGE_INGEST_API_KEY. Non-gateway callers cannot reach the engine.

4. Worker-side ownership re-check + per-id mutex

   Update / delete / clear_all do GET /memories/{id} → verify user_id == scope.userId → mutate, wrapped by a KV mutex (SETNX svc_mutex:mem:{id} 2-second TTL) so concurrent Worker requests on the same id serialize. Eliminates the TOCTOU race that retry-on-fetch would otherwise expose. Cross-user id returns not_found.

5. Worker is the sole billing emitter

   Every per-user usage event to payment-gateway originates from the Worker. The engine never reaches payment-gateway and only sees the user_id string as request-body metadata. A Railway env leak (GOOGLE_API_KEY_MEMORY or MEM0_ADMIN_API_KEY) lets an attacker hit Google and our engine until we rotate, but cannot mint usage events for arbitrary users.

Mem0 itself partitions by user_id in its pgvector schema; every query requires at least one of user_id / agent_id / run_id and refuses to run without it. We rely on user_id only.

Hardening Before External Users

V1 ships with internal users only. Before flipping the switch to external, these items must land. They are not blocking the initial implementation but ARE blocking the external GA gate.

1. Audit log on destructive ops

   V1 has none — debugging "where did my memory go?" requires SSH-ing into Railway. Before external users, add a Worker-side audit table (one row per update/delete/clear_all) with nightly export to R2 object-lock for HIPAA/SOC2-Type-II evidence.

2. Postgres-backed history

   Mem0 history currently disabled (":memory:"). Before external users, a ~50 LOC custom PostgresHistoryManager subclass restores content-diff forensics without re-introducing a SQLite volume.

3. Eval recall@5 stable for 14 days

   At staging-traffic volume; size the eval set to detect a 10pp regression with significance.

4. Tighter DR posture

   Either move Postgres backups to WAL-G shipping WAL to R2 every 5 min (5-min RPO, off-Railway redundancy), or wait for Railway's native PITR if/when it ships. v1's 24-hour Railway-native RPO is below the bar for irreversible personal-memory data at external-user scale.

5. Successful DR drill

   RTO ≤ 4 h, RPO meeting the new posture.

What Has No Worker-side Alternative

Two things genuinely cannot be recovered on the Worker side under pure-mem0. Calling them out explicitly so future revisits know what they cost and how to bring them back.

Everything else (ownership check, per-user usage emission, rate limit, dedupe, recency decay) has a Worker-side alternative — see the Components and Multi-tenancy sections.

Deployment And DR

Required staging validation

1. Round-trip

   remember → search_memory → update_memory → delete_memory.

2. Cross-user attempt

   From a fresh JWT, try to read/update/delete another user's id; must receive not_found.

3. Recreate engine service

   Tear down the FastAPI service while keeping Postgres; redeploy; verify memories survive (no volume to lose).

4. Quarterly DR drill

   Restore latest snapshot + replay WAL into a scratch Railway project; run eval suite; record actual recovery time.

Implementation Libraries

Versions pinned to the latest stable as of 2026-05-18.

ai-gateway (TypeScript, Cloudflare Worker)

Memory engine (Python — zero of our own code)

The engine is the vanilla mem0ai/mem0 Docker image plus an init.sql. We write zero lines of Python.

Testing And Rollout

Eval harness (infra/mem0/eval/golden.jsonl): 50–100 hand-crafted memory/query pairs. CI runs the suite on PRs; recall@5 must not regress by more than 5 percentage points vs main.

1. Provision per env

   Generate MEM0_ADMIN_API_KEY (32-byte random); mint a Google API key (GOOGLE_API_KEY_MEMORY) in a dedicated "memory" GCP project restricted to the Generative Language API; Railway project with Postgres and Railway native daily backups enabled; run init.sql against Postgres; deploy the vanilla mem0ai/mem0 Docker image.

2. Land code and smoke-test dev

   Catalog shows memory; MCP tools list at /engines/memory/mcp; full cycle works; D1 dedupe rows appear; payment-gateway shows one {operation, count:1} usage event per successful MCP tool call (failures emit nothing).

3. Promote to staging via main push

   Repeat smoke + DR drill.

4. Production with internal users

   Gate external rollout on the tighter DR posture from Hardening landing (WAL-G to R2 or Railway PITR) and eval recall@5 stable for 14 days.