Eight Sources, One Query: Building a Unified Agent Memory

How we built a multi-source knowledge retrieval system by making the tests the spec, and found that scoring calibration — not data collection — is where alignment breaks.

Case Study · Next AI Labs · Human-Agent Research · 12 min read

Eight Sources, One Query: Building a Unified Agent Memory

The human predicted exactly what would go wrong: "If I ask you to build it, somehow you will make so many assumptions that the end result is unusable."

He was right about the risk. We'd seen it before — an agent receives a clear specification, makes reasonable architectural choices, writes clean code, and delivers something that technically works but practically doesn't. The gap between "the code runs" and "the results are useful" is where most agent tooling dies.

But this time we had a countermeasure. Not a better spec. Not more documentation. Ten real queries.

The Problem: Knowledge Fragmentation

We had six places where agents could find context about the codebase:

| Source | What It Holds | How Agents Accessed It | |--------|--------------|------------------------| | Swarm Memory | 529 JSONL entries — agent learnings, patterns, failures | swarm_search() MCP tool | | Intent DB | 72 human-approved UX decisions | Separate MongoDB query | | Agent Tools | 59 registered tools and skills | HTTP API (required admin auth) | | UX Assignments | 139 verification tasks with commit refs | No agent access | | Git Commits | Months of commit messages | git log (manual) | | Dev Tasks | 186 development tasks | No agent access |

Plus two more we discovered along the way: 223 Swagger API endpoints and 1,100+ markdown documents with timestamps.

The problem wasn't the data. It was the access pattern. An agent starting a task would search swarm memory — the one source it could easily query — and get 5 results from one perspective. If the answer lived in the Intent DB ("don't hard redirect — use modals") or in a commit message from last week ("already fixed this in session recovery"), the agent would never find it.

This is unified knowledge retrieval as a missing primitive. The agent didn't need eight different search commands. It needed one.

The Prediction That Shaped the Build

The human's warning was specific: don't build this from assumptions. He'd watched agents produce architecturally clean but practically useless search tools before. The scoring would look plausible. The results would be technically valid. But the ranking would be wrong in the ways that matter — surfacing generic tool descriptions above specific intent decisions, or burying recent commits below old swarm memories.

His instruction was to treat this as a measurement problem, not a building problem:

1. Define 10 diverse queries based on real agent work
2. Run each query against swarm-only as a baseline
3. Add sources one at a time
4. Only keep changes that measurably improve results

This is test driven alignment — the tests don't verify that the code runs, they verify that the agent would get the right context. A test passes when the expected knowledge sources appear in the results and the overall candidate count improves over swarm-only.

The Ten Queries

We picked queries that would expose different failure modes:

 1. "firebase email verification spam"        — Recent memory + UX assignment
 2. "blackout mode privacy"                   — Commits + swarm + docs
 3. "premium feature gate click"              — Intent DB (core UX decision)
 4. "cross domain login cookie redirect"      — Deep architectural knowledge
 5. "stop sequences question mark coaching"   — Specific implementation detail
 6. "admin simulator user state"              — Tools registry + swarm
 7. "stripe subscription trial legacy cc_trial" — Critical disambiguation
 8. "siri orb thinking animation"             — Very recent UX change
 9. "beta 50 commitment first contract"       — Strategic intent
10. "session recovery auto create conversation" — Multi-source feature work

Each test case included expectSources — the knowledge sources that should contribute. Query 3 should surface intents (it's a core UX decision). Query 8 should surface recent commits and UX assignments (it's a change from this week). Query 9 should surface strategic intents (it's a pricing decision).

The verdict was simple: for each query, did the unified search hit the expected sources? Did it surface more candidates than swarm-only?

Iteration 1: It Works, But the Ranking Is Wrong

The first implementation searched all six sources in parallel and merged results. Technically correct. Every source returned candidates. But the ranking was broken in two specific ways.

Problem 1: MongoDB text scores flooding the top.

MongoDB's $text index returns a textScore that measures relevance through stemming and term frequency. For the query "stop sequences question mark coaching," a generic tool named "coaching-session-manager" scored 0.98 — because MongoDB stemmed "coaching" to match. The actual best result — a swarm memory about stop sequence implementation — scored 0.23 (cosine similarity).

0.98 beats 0.23. The wrong result wins.

Problem 2: Cosine similarity operates on a different scale.

Vector search returns cosine similarities between 0.0 and roughly 0.3 for relevant results. Text search returns scores between 0.5 and 1.0. Without normalization, vector results are mathematically invisible — they can never outrank even a mediocre text match.

This is the core insight: source authority weighting isn't just about which source matters more. It's about making scores from different retrieval methods comparable. A 0.25 cosine similarity might represent a perfect semantic match, while a 0.95 text score might represent a stemming accident.

Iteration 2: Score Normalization

Two fixes:

Cosine rescaling. We mapped the vector store's natural range (0.0–0.3) to a 0.0–1.0 relevance scale:

const relevance = Math.min(1.0, rawCosineScore / 0.3);

A cosine score of 0.3 or above now maps to 1.0 relevance — giving strong semantic matches a fair shot against text search results.

Keyword validation on text scores. Before accepting a MongoDB text score, we double-check that the actual query keywords appear in the result text:

const queryWords = query.toLowerCase().split(/\s+/);
const docText = (doc.name + " " + doc.description).toLowerCase();
const matchRatio = queryWords.filter(w => docText.includes(w)).length / queryWords.length;
if (matchRatio < 0.3) continue; // Skip stemming accidents

This catches the "coaching-session-manager" problem — MongoDB matched on stemming, but the actual keywords "stop," "sequences," and "question" don't appear in the tool's text.

After iteration 2: 10/10 expected source hits. 10/10 improved over swarm-only.

This is normalized signal in practice. Raw scores from different retrieval systems are not comparable. The normalization layer is where alignment actually lives — not in the data, not in the query, but in the translation between scoring systems.

The Composite Score

Each result gets a composite score from three factors:

composite = relevance * sourceWeight * recencyDecay

Relevance (0–1): Normalized match quality from each source's native scoring.

Source weight: source authority weighting assigns trust based on how the knowledge was created:

| Source | Weight | Reasoning | |--------|--------|-----------| | Intent DB | 1.4x | Human-approved UX decisions — highest authority | | Swarm Memory | 1.2x | Verified agent learnings with context | | API Endpoints | 1.1x | Structural facts about the codebase | | Agent Tools | 1.0x | Registered but not all human-approved | | Markdown Docs | 1.0x | Useful but variable quality | | UX Assignments | 0.9x | Verification tasks, not decisions | | Dev Tasks | 0.8x | Work items, often stale | | Git Commits | 0.7x | Noisy signal, many irrelevant commits |

Intents get 1.4x because they represent explicit human decisions about how the product should behave. When an agent finds both a swarm memory saying "we usually use modals" and an intent saying "session expiry MUST use a dismissable modal," the intent should rank higher. It's not a suggestion — it's a decision.

Recency decay: e^(-0.012 * daysAgo) — an exponential curve where today's results score 1.0, last week's score 0.85, last month's 0.6, and three months ago 0.35. Recent work matters more than historical patterns.

Adding Tier 1 Sources

After the core 6 sources passed all tests, we added two more:

Swagger API endpoints (223 paths): Cached from swagger.json on first search. For a query like "auth register firebase endpoint," this surfaces the relevant endpoint with its summary — context that previously required an agent to manually read the API specification.

Markdown documents (1,100+ files): Indexed via git ls-files "*.md" with heading extraction and modification timestamps. This surfaced a result we didn't expect — for the query "blackout mode privacy," a markdown audit report scored #1 overall, above all swarm memories and commits. The document was a comprehensive privacy audit that no other source had indexed.

Both additions passed the same 10-query test suite with no regressions. The doc source appeared in 3 of 10 tests, contributing genuinely useful context each time.

Before and After: The Same Query, Two Worlds

Query: "premium feature gate click"

Swarm-only:

5 candidates, all from swarm memory
Top result: "When skip-thinking prototype removed" (score: 0.30)
Sources: swarm

The top result is tangentially related at best. The agent would need to search separately — maybe the Intent DB, maybe the tools registry — to find the actual answer.

Unified (agent_find):

38 candidates from tools, UX assignments, and intents
Top results:
  1. how-to-gate-premium-features-with-simulation-support  (tool, 0.72)
  2. Model selector gates premium models for LIMITED_ACCESS  (ux, 0.67)
  3. Trial Tier 1: Full Access (No Gates)                   (intent, 0.66)
  4. Core Platform Feature Access Matrix                     (intent, 0.66)
  5. Step 4: Value Discovery + Feature Gates (Days 1-7)      (intent, 0.66)

The #1 result is a registered skill with the canonical gating pattern. The next four are intents and UX assignments that provide the reasoning behind the pattern. An agent with these five results knows what to build, why it's built that way, and what the exceptions are.

The swarm result didn't even appear in the top 5. Not because it was removed — because better answers exist in sources the agent couldn't previously reach.

The Architecture

agent_find("query")
    |
    +---> searchSwarm()     --> Vector cosine similarity (529 entries)
    +---> searchIntents()   --> MongoDB regex on title/primaryIntent/tags
    +---> searchTools()     --> MongoDB $text + keyword validation
    +---> searchUx()        --> MongoDB regex on summary/commits
    +---> searchCommits()   --> git log --grep (all repos)
    +---> searchTasks()     --> MongoDB $text + keyword validation
    +---> searchSwagger()   --> Cached JSON keyword match (223 paths)
    +---> searchDocs()      --> git ls-files + heading extraction (1100+)
         |
         v
    Normalize scores across retrieval methods
         |
         v
    composite = relevance * sourceWeight * recencyDecay
         |
         v
    Deduplicate by title --> Return top N

All 8 searches run in parallel (Promise.allSettled). A source failure doesn't crash the search — it returns fewer candidates. The swagger and markdown caches load lazily on first use and persist for the session.

The entire system is one function: agentFind(query, { limit, sources, db, vectorStore, repoPath }). It's wired as an MCP tool, so any Claude Code session can call it as agent_find("topic keywords").

What the Test Suite Actually Caught

Three failures, each invisible without real-world test cases:

1. Stemming false positives (iteration 1): MongoDB's text index matched "coaching-session-manager" for the query "stop sequences question mark coaching" with a 0.98 score. The keyword validation filter caught this — only 1 of 5 query words actually appeared in the tool's text.

1. Scale mismatch (iteration 1): Cosine similarity maxes out around 0.3 for strong matches. Text search starts at 0.5. Without rescaling, every text result outranks every vector result regardless of actual relevance. The rescaling formula min(1.0, cosine/0.3) fixed this.

1. Source authority inversion (would have shipped without weights): For "beta 50 commitment first contract," swarm memories about payment configuration (generic) would have outranked the actual strategic intent document (specific, authoritative) without the 1.4x weight on intents.

None of these would have been caught by unit tests. The code was correct. The results were wrong. test driven alignment caught them because the tests measured alignment — did the right answer surface? — not correctness — did the code throw an error?

What We Still Don't Know

Relevance ceiling. Our keyword matching is literal — "premium" matches "premium" but not "paid tier" or "subscription feature." Semantic expansion (synonym matching, embedding-based reranking) would likely improve recall, but we haven't measured the gap. The current system works because our terminology is relatively consistent. It would degrade in a codebase with less naming discipline.

Weight calibration. The source weights (intent: 1.4, swarm: 1.2, etc.) were set by reasoning about authority, not by optimizing against a labeled dataset. We don't know if 1.4 is the right weight for intents or if 1.1 is right for API endpoints. A future iteration could use the test suite to grid-search optimal weights — but that risks overfitting to 10 queries.

Staleness in the cache. Swagger and markdown caches persist for the session but don't detect file changes. If someone adds a new API endpoint or edits a markdown doc, the cache won't reflect it until the next session. For long-running sessions, this could surface stale results.

Coverage gaps. We search 8 sources. We don't search: conversation logs with users, admin system logs (6,279 entries), client memory vectors (436 embeddings from coaching sessions), or changelog entries. Each of these could contribute useful context for specific queries. The cost of adding them is low — the framework handles new sources easily — but we haven't measured their marginal value.

Query complexity. All 10 test queries are keyword-style: 3-5 words describing a topic. We haven't tested natural language questions ("Why does the login flow set cookies before redirect?") or negation queries ("everything about trials except legacy_trial"). The keyword matching approach would likely underperform on these.

These are real open questions, not performative humility. Each one represents a failure mode we haven't tested against yet.

The system now surfaces context from 8 sources through a single agent_find() call. In testing, it improved over swarm-only search on 10/10 diverse real-world queries, with an average of 28 candidates (vs. 5 from swarm alone) drawn from 3-4 different sources per query. The test suite is the spec — any future source addition or scoring change must pass all 10 queries without regression.