The Audit You Can't Outsource

The phrase "design system maturity audit" is on at least four different consultancies' websites this year. We've helped run a few. The pattern is consistent: a team comes in, examines the system, produces a scored report, presents findings, and leaves. The report is true the day it lands. Within a month or two, the report's claims and the system's actual state have drifted apart, and the team that commissioned the audit has lost the thread on which findings still apply.

This isn't anyone's fault. The audit gets scoped as a snapshot, and a snapshot is what you get. Nothing in the engagement connects that one reading to the system as it keeps moving.

The same problem applies to a consultancy's own work. Pitches describe capabilities. Case studies describe results. Service catalogs describe what's on offer. Some of these are accurate when they're written. Most of them slowly stop being accurate, because the work moves and the documents don't. A consultancy whose own infrastructure has that problem is not in a strong position to sell the fix to a client.

We spent eleven sessions over a couple of weeks running a deep audit of the technical and research substance behind every HeyHi service. Sixty units of capability across six clusters of work, on a single instrument, with named evidence required for every claim. The output is a working completion picture: what we can deliver today, what we can deliver with caveats, what isn't ready yet, and what the gate is for each.

The audit isn't the interesting part.

The instrument

Auditing six folders of work with one instrument is the half of the problem most people skip. Without a shared rubric, the seventh session's findings are not comparable to the second session's findings, and the picture you build at the end is collage rather than a portrait.

The instrument we used is small enough to describe in one paragraph. Each unit of work gets placed on a five-rung ladder: research exists, a coherent spec exists, an implementation exists, the implementation has been evaluated, the capability is in active use or genuinely offerable. Each rung requires named evidence, a file path or a test result or a deployment fact, and no rung is granted on confidence alone. Each unit also gets a status from a fixed eight-value vocabulary. The output framing matches the question the audit exists to answer: what is finished, what is in progress, what is left to do.

That instrument never changed across the eleven sessions. Comparable findings came out the other end. The picture is honest because the rubric was the same.

What it caught

The audit caught what audits always catch and a few things audits usually miss.

It caught services we describe as deployable that lean on a capability we've specified but not yet built, and marketing language that had run ahead of what we could actually demonstrate. In one case it surfaced a flat contradiction between an authoritative summary and the underlying record of work, where the summary was wrong and the record was right.

The thing it caught that we weren't expecting is more interesting. The surfaces that drifted weren't the work itself. The work was mostly current. What had rotted were the summary documents that paraphrase the work, for clients, for the website, for internal navigation. The records held; the things that described them to other people did not. Most of the downstream surfaces we checked lagged the record by weeks.

What the audit was actually for

The picture is useful. The follow-up work the picture surfaced is useful. The instrument is portable, and we'll use it on client engagements where the same shape applies.

What the audit was actually for is something we're still building. Finding that summary surfaces consistently rot while the underlying work holds is a structural observation, not a moral one. Telling people to keep the summaries updated does not work. The fix that does work is making the comparison between the underlying work and the summary surfaces a routine check that runs without anyone having to remember it.

We've specified that fix and partially built it. The part that runs without human attention is in production now. The parts that require careful changes to a piece of our own working infrastructure are queued, with the changes designed and a plan for landing them. The audit's actual deliverable, once the queued changes ship, will be a mechanism that catches the same drift on every future session.

A one-time audit can't make that move. The report is true the day it lands, and the rot starts the next morning. Changing the structure that produces the audit, rather than the audit itself, keeps the picture honest over time.

What this changes about a client engagement

Every forward check a team runs faces the same direction. Code review, design review, the sprint retro, the quarterly business review. Each asks whether the thing being introduced right now is correct. None looks back at something that was correct when it shipped and quietly stopped being true. A snapshot audit is that same blind spot at a larger scale: it inspects the system once, at delivery, and then nothing inspects it again. What keeps a living reference honest is the review pointed the other way, run not once but at the close of each piece of work, bounded to the short list of surfaces that are supposed to stay true.

If you've ever shipped a deliverable that was correct on the day it landed and stopped being correct shortly after, the question worth asking isn't "did the team do good work?" The question is what in the engagement structure made the work stop being current. Most of the time, the engagement scoped the one-time read and never scoped what happens to it after delivery. The drift was always coming; the engagement structure just ensured no one would notice.

Not every engagement needs a backward reconciliation pass built into it. Some deliverables are meant to be read once and retired: a research note, a board memo, a positioning document used for a single pitch. But for any deliverable that's meant to be a living reference, a design system, a token architecture, a service catalog, a set of operational specs, the engagement has to specify what keeps the reference honest after delivery. Otherwise the deliverable becomes a quiet liability.

For our own work, the answer is now being built into the close of every session. For clients, the equivalent shows up in the engagement scope, not as a separate sale but as part of what "done" means when the deliverable is supposed to live.

The audit we ran wasn't to grade ourselves. It was to find the structural failure pattern in our own work and fix the structure that produced it, rather than ask harder of the people running it. Reading the system once would have named that pattern and left it exactly where it was.