We re-ran 17,886 of our own scores.
17,886 matched.

The question everyone is asking about AI-skill scanners this summer is “can they be trusted?” Trail of Bits built malicious skills that walked past five scanners — under an hour each for three of the four. A measurement study ran seven scanners over the same skills and found they agreed on 0.12% of what they flagged. And in Part 1 of this series we showed our own scanner had blocked 13% of the MCP ecosystem — and almost none of it was dangerous.

So we think “can scanners be trusted?” is the wrong question. Any single scan is a floor, not a verdict — we’ve said that since June. The better question, for anyone publishing trust scores, is narrower and harsher:

If someone re-ran your math on your own stored data, would they get your own answer?

A score you can’t replay is an opinion with a number attached. So we tried to break ours.

The test, and the rule we set before running it

MCPSkills keeps a version archive: every tracked MCP server, skill, and package is observed on a roughly 17-hour cycle — currently 2,631 packages, more than 218,000 consecutive observation pairs across 75 days, with zero calendar days missing. Each observation stores the signals we measured, the score they produced, and the algorithm version that produced it.

In late July we audited that archive the way an acquirer’s diligence team would: six independent measurements, each one re-verified by a second, independent pass at full population — no sampling on the numbers that mattered. And we set the kill rule before measuring: if historical scores failed to reproduce materially, everything else stopped.

Result one: 17,886 of 17,886

We re-derived every historical score that carried its stored inputs — 17,886 records — using one rule: compute as of the timestamp the record itself stores.

• 17,886 of 17,886 re-derived exactly. Mean drift across every time-dependent signal: 0.000.
• Composites recomputed to within 0.01 on 100% of records — 96.5% bit-exact, and the residual is one-decimal rounding on stored signals, not disagreement.
• Every record carries its algorithm version, so engine changes are mechanically separable from real project changes. Only 3.26% of consecutive observation pairs straddle an engine bump.

Result two: the embarrassing one

This series exists because we publish our failures with the same precision as our wins. Part 1’s failure was over-blocking. Part 2’s failure is stranger: we had been underreporting change — against our own interest.

The same audit counted every version, score, and tier transition present in the raw tape: about 35,500 across the full corpus. Our published change ledger held 1,538.

Not because the data was missing — because our publishing rules were too conservative. Two of them were deliberate signal floors we stand behind (tiny score wobbles don’t deserve a feed entry; a GitHub commit hash changing isn’t a “release”). But two were policy mistakes. The biggest: when our engine changed and re-tiered part of the corpus, we suppressed those records entirely, on the theory that engine news isn’t project news. The theory was half right — engine re-tiers don’t belong in your “what changed this week” feed — but suppressing them made the record incomplete. A complete ledger should store everything and label it, and let each surface filter for its audience.

So that’s what we shipped:

• Every change record now carries a cause label — subject_change (the package changed) or algorithm_change (our engine changed) — plus the engine version on both sides of the transition.
• We re-derived the complete ledger from the raw tape: 1,563 missing records recovered, including the entire re-tier wave from Part 1’s scanner overhaul — which means the claims in that published report now cross-reference directly against our own stored data.
• And the replay property held a second way: of the 7,835 change records we had already published, the re-derivation reproduced 7,835 byte-identically. Same inputs, same math, same bytes — months later.

The public feed still shows only subject changes. Your “what changed” page should never be flooded by our engine’s opinions about itself. But the ledger underneath is now complete, labeled, and — the part we care about — checkable.

What this does not prove

Honesty about limits is the point of this series, so:

Why this matters beyond us

The ecosystem is converging on the same lesson from three directions: scanners can be bypassed, scanners disagree, and scanners over-flag. The response can’t be “trust our scanner more.” OWASP’s new Agentic Skills Top 10 points the right way — its first entry asks registries to display publisher trust level and scan status in the UI. We’d add one word to that prescription: display trust signals you can audit. A number on a page is only as good as the record behind it, and the record is only as good as your ability to replay it.

That’s the standard we’re holding ourselves to, in public, with our misses documented in this series. The changes feed is live, the current-scores dataset is CC BY 4.0, the methodology is documented, and if you re-run our math on our data and get a different answer, we want to hear about it — that’s not a threat to the system, it’s the system working.