Mining 100 Conversations to Close the Alignment Gap

How we extracted 175 implicit human preferences from conversation logs and turned them into searchable agent knowledge, reducing assumption cascades to near-zero.

Case Study · Next AI Labs · Human-Agent Research · 10 min read

Mining 100 Conversations to Close the Alignment Gap

When agents repeatedly ask questions you've already answered in previous sessions, the problem isn't intelligence — it's memory architecture.

The Problem

We had a recurring pattern that was quietly expensive: an agent would receive a task like "fix the session expired handling," build a solution that hard-redirected users to /login, and then get corrected. "No — use a dismissable modal. Hard redirects are banned." The agent would fix it, the session would end, and three days later a different agent session would make the exact same mistake.

This is intent drift in action. Each individual agent session was capable and well-intentioned. But human preferences — the accumulation of hundreds of micro-decisions about how the product should feel — lived only in conversation history. They weren't searchable. They weren't indexed. They existed as scattered corrections across dozens of sessions, invisible to future agents.

The cost wasn't dramatic. No outages. No data loss. Just a slow bleed of back-and-forth cycles where humans re-explained things they'd already decided. Three clarifying questions per session that shouldn't have needed asking. A steady tax on trust.

What We Tried

First Attempt: More Documentation

The obvious fix was writing things down. We created a CLAUDE.md file, added rules, documented patterns. This worked for the most critical rules — "never use ambiguous trial" made it into every session's context. But the file grew. The rules that made it in were the ones dramatic enough to remember to write down. The subtler preferences — "admin tooling should never be affected by limited_access tier," "first-time visitors see the coaching taster, not a login prompt" — fell through the cracks because they felt obvious in the moment.

Documentation captures what you think to document. It misses what you correct in passing.

Second Attempt: Mining the Corrections

The insight came from noticing that the corrections themselves were the documentation. Every time a human said "no, not like that — like this," they were expressing a preference with perfect specificity. The conversation logs contained 100 sessions of exactly this: real decisions, real corrections, real preferences — all with enough context to understand when and why they applied.

So we built a preference mining pipeline.

The architecture was simple:

1. Take 100 conversation logs (JSONL files, each containing a full agent session)
2. Deploy 5 parallel mining agents, each processing 20 conversations
3. Extract three categories of knowledge: UX intent statements, workflow patterns, and decision heuristics
4. Merge and deduplicate across all batches
5. Hydrate the results into searchable registries

The extraction prompt focused on three signals:

• Direct corrections: "No, don't do X. Do Y instead." These are the highest-confidence preferences.
• Explained decisions: "We do it this way because..." These carry the reasoning, not just the rule.
• Repeated patterns: The same correction appearing across multiple sessions. Frequency = importance.

The Three-Category Split

Not all preferences are the same shape. We discovered that mined knowledge naturally falls into three categories, each needing a different storage format:

UX Intent Statements (68 extracted): Clear directives about how the product should behave from the user's perspective. "When a session expires, the user should see a dismissable modal with evidence of their prior session, not a hard redirect." These have a when/then structure that maps directly to searchable intent.

Workflow Patterns (44 extracted): Order-of-operations directives for how work should be done. "Fix visible issues first, then root-cause. Each fix gets a separate commit with intent documentation." These are meta-instructions about process.

Decision Heuristics (63 extracted): Rules about when to look deeper, act immediately, or double-check. "Never assume what code does — read it first." These prevent assumption cascade by intervening before the first wrong assumption.

Total: 175 directives, extracted from 177 raw candidates after deduplication. Two duplicates in 177 — a 1.1% overlap rate across 5 independent mining agents, which suggests the extraction prompt was consistent enough to produce similar outputs from different agents seeing different conversations.

The System That Emerged

The mining operation revealed a gap in our infrastructure: we had places to store knowledge (Intent DB, Agent Tools Registry, meta-analysis docs) but no systematic way to fill them from experience. The pipeline we built closes that loop.

Knowledge Hydration Pipeline

This is knowledge hydration in practice. Each category of mined knowledge flows to a different registry:

Conversation Logs (100 JSONL files)
        |
   [5 Mining Agents]  ← parallel extraction
        |
   [Merge/Dedup Agent] ← normalize schemas, remove duplicates
        |
   ┌────┼────────────────┐
   ▼    ▼                ▼
Intent DB   Agent Tools     Meta-Analysis
(68 UX      Registry        Docs (5 reports)
 intents)   (107 tools:
            44 workflows +
            63 heuristics)

Each destination has different query patterns:

• Intent DB: Agents query "what should happen when a user does X?" and get back documented intent with certainty scores
• Agent Tools Registry: Agents search for workflow patterns and decision rules by tag (e.g., tag:heuristic, tag:workflow)
• Meta-Analysis Docs: Comprehensive audit reports that provide deep context when agents encounter complex systems

Concept Atomization

The mining process also demonstrated concept atomization. Rather than storing preferences as monolithic documents, we broke them into atomic units — each with a single directive, a when/then trigger condition, and a source reference. This granularity is what makes the knowledge searchable at decision time rather than just readable as background context.

An atomic directive looks like:

{
  "directive": "Hard redirects to /login are banned — use dismissable modals",
  "when": "Agent implements auth failure handling",
  "then": "Show modal with dismiss option, not a redirect",
  "source": "conversation-mining-feb-2026"
}

How It Works in Practice

Before: The Assumption Cascade

Scenario: Agent receives task "handle expired session for returning users."

1. Agent checks documentation → finds general auth patterns but nothing about session expiry modals specifically
2. Agent assumes hard redirect to /login is standard (reasonable assumption from web development conventions)
3. Agent implements redirect → triggers pre-commit hook → passes (no UX tests for this flow yet)
4. Human reviews → rejects → explains modal requirement → agent fixes
5. Wasted: 1 review cycle, human's context switch, agent's wasted implementation time
6. Learned: Nothing persistent. Next agent session starts from zero.

This is a textbook assumption cascade. The first assumption ("redirect is standard") was individually reasonable. But it cascaded into implementation work that had to be undone.

After: Knowledge-Informed Decision

Same scenario, post-hydration:

1. Agent receives task "handle expired session for returning users"
2. Agent queries Intent DB → finds: mined-hard-redirects-to-login-are-banned (certainty: 35/100, source: conversation mining)
3. Agent queries Agent Tools → finds: heuristic-session-expired-modal-requires-actual-evidence
4. Agent builds dismissable modal with session evidence check → matches human intent on first attempt
5. Saved: The review cycle, the context switch, the trust erosion

The certainty score of 35/100 is deliberate. These are agent-proposed items capped at 50 to signal "this came from mining, not from explicit human documentation." When a human approves them, the score rises. This creates a natural alignment feedback loop: mined preferences start as low-confidence suggestions, gain confidence through approval, and eventually become high-confidence rules.

What We Learned About Defensive UX

A recurring theme in the mined directives was defensive ux coding. The human's core principle: code should default to the best possible user experience when internal systems fail.

The canonical example: don't check "is this user paying?" and restrict features on failure. Instead, check "should this user be limited?" — so if the check fails, paying users get full access rather than being wrongly restricted.

This principle appeared 6 times across 100 conversations in different forms. Each time, the correction was specific to a different feature. But the underlying principle was the same. Mining surfaced this pattern; atomization made it searchable; hydration made it available at the moment an agent writes a feature gate.

What We Still Don't Know

Scale questions: This pipeline processed 100 conversations in approximately 8 minutes using 5 parallel agents. We haven't tested it at 1,000 or 10,000 conversations. The deduplication logic (first 60 characters of directive text) may need refinement at higher volumes.

Certainty calibration: Agent-proposed items cap at certainty 50. But we don't yet have data on how often mined preferences turn out to be wrong — cases where the human's in-the-moment correction was itself a mistake they'd want to undo. The approval step catches this, but we haven't measured the false-positive rate.

Concept drift: Preferences evolve. A directive mined from a conversation 3 months ago may no longer reflect current thinking. We don't have a staleness mechanism yet — no way to mark a mined preference as "superseded" by a newer one without manual review.

Cross-domain transfer: All 175 directives came from one product (IX Coach). We don't know if the mining prompt generalizes to other codebases, or if the three-category split (intent/workflow/heuristic) is universal or specific to our development style.

These are real open questions, not performative humility. Each one represents a failure mode we haven't tested against yet.

The mined directives now live in three searchable registries: an Intent DB for UX decisions (68 entries with certainty scores), an Agent Tools Registry for workflow patterns and decision heuristics (107 entries), and a set of meta-analysis documents providing deep context for complex systems. Each is queryable at the moment an agent needs to make a decision.