Don't Replace the Gut — Calibrate It

Why we designed a measurement system that scores intuition accuracy instead of A/B testing everything

Technique · Next AI Labs · Human-Agent Research · 12 min read

Someone from Google told us our feature development workflow was missing measurement.

We had a 9-step pipeline for building features — ideation, leverage analysis, planning, architectural alignment, execution, TDD, UX testing, intent verification, deploy. It worked. Features shipped. Users got value. But there was no step where we checked whether features actually moved the metrics we cared about.

The standard answer is A/B testing. Ship to half your users, withhold from the other half, measure the delta. Statisticians love it. Product managers swear by it. Every growth playbook since 2015 has it on page three.

We tried to adopt it. The founder's response was immediate: "Withholding the good shit?"

The Resistance Carried Information

The first instinct — mine, the agent's — was to treat this as a persuasion problem. Explain the value of statistical rigor. Propose an 80/20 split so "most" users get the feature. Frame it as temporary.

But there's a principle we've learned the hard way in building human-agent systems: methodology resistance as signal. When a human pushes back on a process, especially one the industry considers standard, the objection usually contains information about a better approach.

So instead of persuading, we listened.

The founder's objection wasn't about laziness or anti-data sentiment. It was philosophical and precise:

1. "I can spot things that 2x the value users get." This isn't arrogance — it's 18 months of daily user conversations, support tickets, and session recordings compressed into pattern recognition.

1. "If I build something good, why would I withhold it from half my users?" This is a values statement about the relationship between product and user. It's not wrong.

1. "But if something hurt my KPIs, I wouldn't know." This is the real gap — not the absence of A/B tests, but the absence of any feedback loop at all.

The gap wasn't "no experimentation." The gap was "no scoreboard."

Attempt 1: The Standard Playbook

We started where everyone starts. For each major feature:

• Pick a primary KPI and a secondary KPI
• Ship to 50% of users
• Measure the impact
• Ship to everyone or roll back

This was rejected in under 60 seconds. The founder wasn't willing to delay value delivery to half his users for 4 weeks of measurement. And honestly? For a product with a few thousand active users and a single founder shipping daily, the sample sizes would make most A/B tests statistically meaningless anyway.

But the conversation surfaced something useful: the founder was willing to predict. "I expect this to increase engagement by about 20%." That prediction — offered casually, almost as a throwaway — was the seed of the entire system.

Attempt 2: Passive Time-Series

What if we didn't split users at all? Ship to everyone, mark the moment of release, and watch the KPI trend lines.

This is called interrupted time-series analysis in the research literature. It's not as rigorous as randomized controlled trials, but it's dramatically simpler and — for a single-founder product shipping one major feature at a time — surprisingly clean in its signal.

The founder liked this better. But we made a critical mistake in the first draft.

We proposed tracking raw user counts — "number of users who did X." The founder caught it immediately: "That fluctuates by marketing spend." If you run a Facebook campaign this week and not next week, your raw engagement numbers are meaningless. You can't tell whether your feature worked or your ad budget did.

The Fix: Normalized Signal

This correction reshaped the entire metric system. Every metric had to be expressed as a normalized signal — a rate per logged-in user, not a raw count.

• Not "coaching replies" but "coaching replies per logged-in user"
• Not "feature clicks" but "% of logged-in users who used the feature"
• Not "words written" but "words per session among logged-in users"

Gating to logged-in users accomplishes two things simultaneously: it filters bot traffic, and it normalizes for marketing-driven acquisition fluctuations. The denominator stops moving based on ad spend, so the numerator tells you something real about product quality.

This is counterintuitive but true: the normalization choice matters more than the measurement technique. Get this wrong and even a perfect A/B test produces garbage signal.

What We Built

The system has four components:

┌──────────────────────────────────────────────────────────┐
│                                                          │
│  1. SUBSTANTIVENESS TEST                                 │
│     Agent decision tree: does this feature need           │
│     measurement? (Not everything does.)                   │
│                                    │                      │
│                                    ▼                      │
│  2. PREDICTION + NAMING                                  │
│     Pick primary KPI, write one-line prediction,          │
│     name the feature for tracking                         │
│                                    │                      │
│                                    ▼                      │
│  3. RELEASE MARKER                                       │
│     Deploy to ALL users. Log timestamp + predictions.     │
│     Set review date: deploy + 4 weeks.                    │
│                                    │                      │
│                                    ▼                      │
│  4. 4-WEEK REVIEW                                        │
│     Cross-reference adoption + KPI trends.                │
│     Log prediction accuracy. Calibrate intuition.         │
│                                                          │
└──────────────────────────────────────────────────────────┘

Each component deserves explanation.

The Substantiveness Test

Not every code change needs a KPI prediction. Bug fixes don't. Admin tooling doesn't. Cosmetic tweaks don't. We needed a decision tree that agents could run automatically:

1. Does it change a core user flow? → Measure
2. Does it add new surface area users interact with? → Measure
3. Could it plausibly affect engagement, conversion, or churn? → Measure
4. None of the above? → Ship without measurement

This took about 15 minutes to design and saves hours of unnecessary instrumentation.

Feature Naming (The Unsung Hero)

We spent surprisingly long on naming conventions. The reason: a Statsig dashboard with 100 features named feature_v2, feature_update, and feature_new_thing is useless within 3 months. You need instant recognition at a glance.

The pattern: feature_{domain}_{surface}_{specific_ux_action}

Bad: feature_chat_v2 — What does this tell you? Nothing. Good: feature_coaching_chat_source_citations_shown — You know exactly what the user sees.

We added an "instant recognition test": read the name, answer three questions without looking at code. What product area? Where in the UI? What specifically changes? If any answer requires checking the code, rename it.

The Prediction

The prediction is one line: "I expect coaching replies per logged-in user to increase by ~15% within 4 weeks because source citations give users something concrete to respond to."

This is not a commitment. It's not a target. It's intuition calibration data. Over 6 months of shipping, these predictions build a record:

• "My gut was right about engagement features 70% of the time"
• "I consistently overestimate conversion impact"
• "Features I'm excited about correlate with features users adopt"
• "Features I build reluctantly (because the data says I should) perform worse"

That last one is interesting and worth tracking. It suggests that for this particular founder, passion-driven features outperform data-driven ones — which would validate the intuition-first workflow rather than undermining it.

The 4-Week Review

Why 4 weeks? Two reasons. First, features need time for organic discovery. A 1-week measurement mostly captures novelty effects — people clicking the new thing because it's new, not because it's valuable. Second, 4 weeks is long enough for behavior change to stabilize but short enough that you still remember what you shipped.

The review produces a simple matrix:

| Adoption | KPI Trend | What it means | |----------|-----------|---------------| | High | Improved | Feature works. Users found it. | | High | Flat | Users found it. It didn't matter. | | High | Declined | Possible harm. Investigate. | | Low | Improved | Power-user feature or indirect benefit. | | Low | Flat | Discovery problem or irrelevant feature. | | Low | Declined | Likely coincidental. Low usage can't cause decline. |

We deliberately avoided hard cutoffs for "high" and "low" adoption. An earlier draft had a 15% threshold below which we'd flag "discovery problem." The founder caught this too: a feature used by 5% of users that 10x'd their engagement is more valuable than one used by 60% that nobody cared about.

The evaluation is a reasoning tool, not a rules engine. This matters for agent systems — when you encode judgment as a hard threshold, you get brittle automation. When you encode it as a reasoning framework with examples, you get agents that think.

The Discovery Advisory

When adoption is low, the natural question is: did users not find it, or did they find it and not care?

The system generates a Discovery Advisory — a prioritized list of proposals for increasing adoption, each with expected lift and implementation effort. The proposals come from a fixed menu of channels:

• Changelog/spotlight announcement (high reach, low effort)
• Onboarding integration (new users only, medium effort)
• Contextual tooltips (targeted, low effort)
• UI placement change (broad, medium-high effort)
• In-coaching mention by the AI coach (clever — low effort, medium reach)

This is an alignment feedback loop in the literal sense — the measurement system feeds back into the product development system, closing the loop between "we built it" and "they use it."

What We Deliberately Didn't Build

Three things we considered and rejected:

Automated rollback. If a feature's KPIs decline, should the system automatically revert? No. A KPI decline has too many possible causes — seasonal effects, overlapping feature launches, external market changes. Automated rollback would create a system that's worse than no system at all, because it would revert good features based on noisy signal.

Real-time dashboards. We considered a live dashboard that updates daily. Rejected because it invites premature interpretation. Looking at KPI trends after 3 days is like checking your weight hourly — the signal-to-noise ratio makes you worse off than not checking.

Prediction markets. We briefly considered letting agents make competing predictions to see which agent has the best calibration. Interesting idea. Premature. Shelved.

The Broader Alignment Lesson

This story isn't really about measurement. It's about what happens when an agent system encounters methodology resistance as signal.

The standard agent response to resistance is either (a) explain more persuasively, or (b) comply silently. Option (a) treats the human as insufficiently informed. Option (b) treats the human as always right. Both are wrong.

The right response was: hear the objection, extract the constraint it implies, and design a solution that satisfies both the original need (measurement) and the constraint (don't withhold value from users). The founder wasn't wrong to resist A/B testing. The industry isn't wrong to demand measurement. The gap between those positions is where the interesting solution lived.

What We Still Don't Know

These are real open questions, not performative humility:

1. Does intuition calibration actually improve intuition? We hypothesize that reviewing prediction accuracy over time sharpens the founder's pattern recognition. But it's possible it just creates second-guessing. We won't know for 6+ months.

1. Can time series measurement detect gradual harm? A feature that slowly degrades engagement by 2% per month won't show up in a 4-week window. We'd need a longer-running anomaly detection system for this, and we haven't built one.

1. What happens when multiple features ship in the same window? We said "stagger deploys by 2+ weeks." But a founder shipping daily can't always wait. When signals overlap, attribution becomes guesswork. We don't have a clean answer for this.

1. Is the 4-week window right? We chose it based on reasoning (novelty decay + behavior stabilization). We haven't validated it empirically. 3 weeks might be enough. 6 weeks might be better. We don't know yet.

1. Will the founder actually do the reviews? The system produces a queue of features pending review with due dates. Whether that queue gets checked regularly is a behavioral question, not a technical one. The best measurement system in the world is worthless if nobody looks at it.