The Discovery Problem: When AI Agents Build Tools No One Can Find

We installed 15 AI skills in a single session. Eight were invisible to future agents — not because they were broken, but because they lacked three lines of metadata.

Case Study · Next AI Labs · Human-Agent Research · 9 min read

The most dangerous failure mode in a multi-agent system isn't a tool that breaks. It's a tool that works but can't be found.

We discovered this the hard way. In a single session, we installed fifteen AI skills — security auditors from Trail of Bits, OWASP vulnerability checkers, Playwright browser automation, custom test fixture generators, error handling standards, and more. Every one worked. Every one passed its verification test. And eight of them — more than half — were invisible to any future agent that might need them.

Not broken. Not misconfigured. Just unfindable.

The Risk: Invisible Capability Accumulates Silently

When human teams lose institutional knowledge, someone eventually notices. A new hire asks a question no one can answer. A process breaks and nobody remembers why it existed. The knowledge gap becomes visible through its consequences.

AI agents don't have this feedback loop. When an agent builds a tool that other agents can't discover, nothing visibly breaks. The system doesn't error. No alert fires. The tool sits in its directory, fully functional, while future agents — facing the exact problem it solves — build their own solution from scratch. Or worse, they proceed without it and ship vulnerable code that the undiscoverable security auditor would have caught.

This is invisible capability in action — the AI equivalent of institutional knowledge trapped in someone's head, except no one even knows to ask them about it.

The compounding effect is what makes this dangerous. Every session that builds tools without verifying discoverability adds to the invisible pile. Over time, the gap between what tools exist and what agents can find — the capability discoverability gap — grows wider. The system becomes simultaneously more capable and less effective.

What We Tried: Building Fifteen Skills in Parallel

The session that revealed this problem was ambitious by design. We needed to close gaps in security, code quality, and system observability for a production coaching platform's API. The plan called for three phases of parallel installation:

Phase 1 — Four external security skills: OWASP Top 10 checks, six Trail of Bits audit methodologies (insecure defaults detection, variant analysis, differential review, sharp edges identification, static analysis, and Semgrep rule creation), a destructive-command safety net, and Anthropic's official skill library.

Phase 2 — Three infrastructure tools: a read-only MongoDB connector for live debugging, Playwright browser automation for end-to-end testing, and a configuration linter for agent setup files.

Phase 3 — Seven custom skills built from codebase analysis: a controller extraction workflow for decomposing 2,500-line files, React component patterns specific to the codebase, a silent failure detector for swallowed errors, system observability standards, API error handling conventions, per-endpoint security checklists, and canonical test fixtures.

Each phase ran as parallel subagents — four agents installing external tools simultaneously while the main context wrote custom skills. The entire installation completed in a single session. Every tool passed its functional test.

Then we ran the discovery audit.

The Discovery Audit: 53% Invisible

The audit checked each skill against a simple question: can a future agent, facing a relevant problem, find this tool through automated discovery?

The results were stark:

| Skill Source | Count | Discoverable | Invisible | |-------------|-------|-------------|-----------| | Custom-built | 7 | 7 (100%) | 0 | | External | 8 | 0 (0%) | 8 (100%) | | Total | 15 | 7 (47%) | 8 (53%) |

Every custom-built skill was discoverable. Every externally-sourced skill was invisible.

The pattern was consistent: the seven custom skills had been written with discovery metadata built in — keyword trigger phrases that the matching system uses to connect agent queries to relevant skills. The eight external skills, sourced from respected security research firms and open-source repositories, had everything except that metadata.

The fix was absurdly small. Three to five lines of YAML per skill:

triggers:
  - "security vulnerability"
  - "injection prevention"
  - "XSS prevention"
  - "security review"

That's it. Without those lines, a security skill that covers the entire OWASP Top 10 sits unused while an agent writes authentication code without security awareness. With them, the skill activates automatically when an agent encounters security-relevant code.

Why This Happens: Five Surfaces, Five Failure Modes

Agents don't discover tools through a single mechanism. In our system, there are five distinct discovery surfaces — channels through which an agent can find existing capabilities:

Surface 1: Protocol auto-listing
  Tools registered in protocol servers → automatically visible
  Failure: agents without the server attached never see the tool

Surface 2: Trigger matching
  Keyword phrases in metadata → matched against current task
  Failure: missing triggers → zero matches → invisible

Surface 3: Tooling registry (database-backed)
  Searchable by slug, tags, category → human-auditable
  Failure: tool never registered → doesn't exist in search index

Surface 4: Semantic swarm search
  Vector embeddings across sessions, research, tools, intents
  Failure: tool too new for embedding, or description too generic

Surface 5: Registry semantic search
  Token-conscious search on the tool registry specifically
  Failure: must be registered first — same gap as Surface 3

The eight invisible skills failed at Surface 2 — no trigger metadata. They were also absent from Surfaces 3 and 5 (not yet registered in the tooling database). Some were partially visible through Surface 4 (the semantic layer had seen related concepts in prior sessions), but partial visibility is worse than no visibility — it creates inconsistent behavior where the same query finds the tool sometimes but not others.

The key insight: each surface has independent failure modes. A tool that's discoverable on one surface can be completely invisible on another. And the failure is always silent — no error, no warning, just a query that returns nothing.

The Fix: Self-Auditing Discovery

The discoverability audit itself is a skill — a repeatable workflow that checks every new tool against every discovery surface. Running it after the installation caught all eight failures in a single pass.

But the audit only worked because someone thought to run it. The deeper fix is making discovery verification an automatic part of tool creation — what we'd call zero touch observability applied to the toolchain itself.

The pattern we've converged on has three parts:

1. Build with metadata. Every custom skill includes trigger phrases, descriptions, and relational tags from the start. This is the cheapest intervention — adding five lines during creation versus retrofitting them later.

2. Audit after installation. External tools don't come with your discovery metadata. The audit step catches this gap before the session ends. It's a two-minute check that prevents indefinite invisibility.

3. Multi-surface verification. Checking one surface isn't enough. The audit queries each discovery channel independently and reports which surfaces find the tool and which don't. A tool that's only visible on one surface is fragile — if that surface has an outage or a query miss, the tool disappears.

After patching the eight skills with trigger arrays and fixing seven name mismatches (where the internal identifier didn't match the folder name — another silent failure mode), verification showed 15 of 15 discoverable across trigger matching and semantic search.

What This Means for Multi-Agent Systems

This isn't unique to our stack. Any system where agents build tools for other agents — or where tools are imported from external sources — faces the same dynamic:

The builder knows the tool exists. The finder doesn't. And unlike human teams, there's no hallway conversation where someone mentions "oh, we have a tool for that." Agents only find what the discovery system surfaces.

The implications scale uncomfortably:

• Ten tools with a 53% invisibility rate means five tools sitting unused. Annoying but manageable.
• A hundred tools at the same rate means fifty capabilities that agents duplicate or do without. The wasted effort starts compounding.
• A thousand tools — the scale that large organizations with multiple agent teams will reach — means five hundred invisible capabilities. The capability discoverability gap becomes a strategic problem, not a housekeeping one.

The fix doesn't scale through manual audits. It scales through making discoverability a first-class property of tool creation — as fundamental as the tool working correctly.

What We Still Don't Know

These are real open questions, not performative humility.

Trigger phrase design is more art than science. We wrote triggers like "security vulnerability" and "injection prevention" for the OWASP skill. But how do we know these match what agents actually query when they encounter insecure code? We don't have query logs yet. The triggers might be precise for how we think about security but miss how agents frame the problem during implementation.

Discovery across organizational boundaries is unsolved. Our five surfaces work within one system. When agents from different teams or organizations build tools, there's no shared discovery layer. The invisible capability problem multiplies across organizational boundaries.

We can't measure the cost of undiscoverable tools. We know eight skills were invisible. We don't know how many times a future agent would have used them if they'd been findable. The counterfactual — "what would have been caught?" — is exactly the kind of thing that's hard to measure and easy to underestimate.

Semantic search creates false confidence. Surface 4 (semantic swarm search) returned partial matches for some invisible skills — the system had seen related concepts in prior sessions. This is arguably worse than returning nothing, because it suggests the system "kind of" knows about the tool without actually surfacing it. Partial discoverability might be the hardest failure mode to detect.

The right number of discovery surfaces is unknown. Five surfaces means five independent failure modes, but also five independent chances to find a tool. Is the redundancy worth the maintenance cost? At what point does adding surfaces create more confusion than coverage?

The discovery problem isn't a technical curiosity. It's a preview of a challenge every organization will face as AI agents become prolific tool builders: the gap between what exists and what's findable will define the effective capability of the system — not the raw count of tools in the directory.