Why do documentation initiatives fail within a quarter?
Key Takeaways
For busy support leads: you can find your entire runbook backlog in thirty minutes without a survey, a workshop or a new tool. List every escalation from the last month. Next to each one, write the name of every person who could have handled it alone. Circle the ones with a single name. That circled list, ordered by how often the scenario recurs, is exactly what to document and in exactly what order. Everything else can genuinely wait, and most of it should.
- 1Document the second occurrence, not the first. The first may be a one-off; the second proves recurrence and repays the writing.
- 2Every article costs you forever. Two hundred articles at ten minutes a quarter consumes over a hundred hours a year.
- 3Delete by default. If nobody claims an article at review, archive it instead of rewriting it.
- 4Judgment calls need thresholds. "Use your discretion on refunds" is not a policy; a table with numbers in it is.
- 5Internal docs are not bot content. Runbooks contain known bugs, workarounds and account exceptions you never want quoted to a customer.
Because they invert the natural order. Somebody blocks out a week, the team writes forty articles from memory, everyone congratulates each other, and then the product changes. Six weeks later a new agent follows an article that is now wrong, gets burned, and quietly stops trusting the whole library. After that, nobody reads it, so nobody notices the rest going stale.
Documentation written outside the moment of use has two structural weaknesses. It is written from what the author remembers rather than what actually happens, which means it omits the awkward step everyone stumbles on. And it has no reader waiting for it, so no feedback ever arrives to correct it.
The alternative is unglamorous. Somebody solves a hard problem, spends eight minutes writing down what they actually did, and the next person to face it reads it within a fortnight. That article gets corrected because someone used it. This is the whole difference, and no tool creates it.
What is the write-it-the-second-time rule?
Do not document a scenario the first time it appears. Document it the second time.
The first occurrence proves nothing about recurrence. Support work is full of genuinely singular events, a misconfigured account, an odd browser, a customer doing something nobody will do again, and documenting all of them produces a library where the useful pages are buried among one-offs. The second occurrence is evidence, and it also gives you two examples, which usually reveals which parts of the procedure are stable and which were incidental.
There is a practical benefit beyond volume control. Whoever hits the scenario second is the right author, because they just searched for it, failed to find it, and knows exactly what they wished had been written. That person writes a better article than the original solver would have, and they write it with a reader in mind, namely their past self twenty minutes ago.
How do you find the knowledge only one person has?
Run the bus-factor audit described above, then read the result honestly. Every scenario with exactly one name is a dependency you are carrying without having decided to.
Rank the circled items by two things: how often the scenario recurs, and how bad it is if it goes unhandled for a day. A weekly scenario with one owner is urgent. A twice-yearly scenario with one owner is a note in a shared doc, not a project.
The second free data source is every new hire. Ask them to log every question they had to ask a colleague during their first two weeks, one line each, no formatting. That log is your gap list produced at zero cost, and it has a short shelf life, because the moment they stop being new they stop noticing what was missing. Most teams waste this. It is the single cheapest audit available and it arrives with every hire.
What does a runbook look like when it is actually usable?
Agents read internal docs mid-conversation, with a customer waiting. That constrains the format more than any style guide: scannable, decision points explicit, exit conditions stated, and a defined output at the end. Here is a real one rather than a description of one.
Scenario: customer reports webhooks are not arriving.
Gather first: account ID, endpoint URL, the time window, one example event ID.
- Check delivery logs for that endpoint across the window. Zero attempts means a subscription or filter problem, so jump to step 4. Attempts present, continue.
- Read the response codes. A 4xx means their endpoint rejected us: reply with the exact code and response body, and stop, this is not our bug. A 5xx or timeout means their server was failing: reply with timestamps and our retry schedule, and stop.
- Attempts logged with no response recorded at all suggests our delivery worker is backed up. Check queue depth. Above normal, escalate to engineering with the metric attached and tell the customer we are investigating on our side.
- Confirm the event type is actually subscribed and no filter excludes it. This is the most common cause and the fastest fix.
Escalate immediately, skipping everything above, if more than one account reports the same window, or the customer is mid-migration.
Expected output: either a reply naming the failing side with evidence, or an engineering ticket containing account ID, endpoint, window, example event ID and queue depth.
Notice what makes it usable. Every branch terminates. Two steps say "stop", which prevents the most common runbook failure, an agent following all seven steps when step two already answered the question. And the expected output is stated, so the agent knows when they are finished.
How do you document a judgment call?
With numbers. "Use your judgement on refunds" is not guidance, it is an invitation to interrupt a team lead, and it guarantees two agents give different answers to the same customer in the same week.
The table below uses illustrative thresholds. Yours will differ, and that is fine. The point is that a table exists and contains actual numbers instead of adjectives.
| Situation | Under $50 | $50 to $500 | Over $500 |
|---|---|---|---|
| Our outage or bug caused it | Refund, no approval | Refund, note it in the thread | Refund, tell a lead same day |
| Forgot to cancel, one cycle | Refund, no approval | Refund once per account | Lead approves |
| Forgot to cancel, several cycles | Credit, not refund | Lead approves | Lead approves |
| Change of mind, outside policy | Credit or decline | Decline, offer a downgrade | Decline, offer a call |
| Chargeback history or abuse signals | Decline and log | Decline and log | Decline and escalate |
Publishing this changes two things immediately. Agents stop queueing outside a lead's calendar for routine decisions, and customers stop getting different answers depending on who picked up the thread. It also makes exceptions visible, because an exception now has a rule to be an exception to.
Do the same for your other recurring judgment calls: when to escalate to engineering versus investigate further, when to offer a call, when to fire a customer. Each one is a table, and each table removes a recurring interruption.
What does your documentation actually cost to keep?
Nobody publishes this number, which is why teams keep writing. Every article carries a permanent review obligation, and the total compounds quietly.
| Corpus size | Quarterly review at 10 min each | Per year |
|---|---|---|
| 40 articles | 6.7 hours | about 27 hours |
| 100 articles | 16.7 hours | about 67 hours |
| 200 articles | 33.3 hours | about 133 hours |
| 500 articles | 83.3 hours | about 333 hours |
Ten minutes per article is generous for a quick accuracy check and optimistic for anything needing verification against the live product. Even at that rate, a five-hundred-article internal library consumes over eight working weeks a year of somebody's time, which is why five-hundred-article libraries are almost always out of date. They were never resourced, only written.
The conclusion is a cap, not a warning. Decide what review capacity you actually have, divide by ten minutes, and that is your maximum sustainable corpus. Publishing beyond it is choosing to have wrong documentation, whether or not anyone says so out loud.
Why should deletion be the default review outcome?
Because review cycles that can only result in "update" or "keep" always grow the corpus, and growth is the thing killing you. Flip the default. At review, an article survives only if a named person claims it and confirms it is still accurate. Unclaimed articles get archived, not rewritten.
This feels reckless the first time and stops feeling that way immediately. An article nobody will claim is an article nobody uses, and an unused article is not neutral: it competes in search results with the correct one, and it is the article a new agent finds and follows.
Archive rather than delete, so recovery is trivial and nobody has to be brave. Then watch what gets pulled back out. In our experience it is almost nothing, which tells you what the archived material was worth. Wrong internal documentation is worse than none, because an agent with no document asks a colleague, while an agent with a wrong document confidently tells a customer something untrue.
Should your AI assistant read your internal runbooks?
Not the customer-facing one. This is the mistake we see teams make with the best intentions, and it is difficult to reverse once it has happened.
Internal documentation contains things written for colleagues: known bugs engineering has deprioritised, workarounds that hint at unpatched behaviour, pricing exceptions granted to specific accounts, other customers named in examples, and blunt internal language about product limitations. A retrieval system does not know any of that is sensitive. It sees relevant text and quotes it, and a model asked directly whether a problem is a known bug will find the runbook that says exactly that.
The liability question is settled enough to plan around. When Air Canada's chatbot stated a policy that contradicted the airline's own website, the tribunal rejected the submission that the chatbot was "a separate legal entity that is responsible for its own actions" and held that "it makes no difference whether the information comes from a static page or a chatbot" (Moffatt v. Air Canada, 2024 BCCRT 149). Anything a retrieval system can reach is something you have published, whatever folder it lives in.
The workable arrangement is two corpora with different audiences. An agent-facing assistant behind authentication can read runbooks and is genuinely useful, since it turns lookup time into seconds mid-conversation. The customer-facing assistant reads published articles only. If you do connect internal content to any assistant, write it knowing a model may quote it verbatim, which is a real constraint on how candidly people will write, and one worth naming before you make the change.
How do you know if it is working?
Time-to-competency for new hires is the honest measure, but resist the urge to compare it to a published benchmark. Define it locally: the date a new agent handles a full day unassisted while asking no more than some number of questions you choose. Then compare each hire against your own previous hire, since that is the only comparison where the variables are close to controlled.
Two secondary signals cost nothing. The count of questions in your team chat that an existing document already answers, which measures discoverability rather than coverage. And the share of escalations that turn out to be scenarios you have documented, which measures whether anyone reads the documentation under pressure.
That first signal deserves attention, because it is commonly misdiagnosed. When people keep asking questions your library already answers, writing more articles makes it worse. The problem is search, structure or naming, and the fix is to consolidate and retitle rather than to produce.
Where this breaks
Under about five agents, shared context is real and documentation is largely overhead. Write down only what would be lost if one specific person left tomorrow, and skip the rest without guilt.
In high-turnover environments, outsourced teams especially, the constraint is training throughput rather than document quality. Structured onboarding, shadowing and quality review move the needle; another fifty articles do not.
If your product ships significant change weekly, document only the stable layer. Procedures for a workflow that is being rewritten this quarter will be wrong before they are read.
And to name our own boundary: Corebee is a support platform with a knowledge base, not a document management system. If you need approval workflows, versioned sign-off, retention policies or compliance attestation on documents, buy a tool built for that and keep it separate from support content. Where we are useful is keeping the conversation, the article and the escalation path in one place, at $99 a month flat so putting every teammate in the same inbox is not a budget conversation.
What should you do in the first two weeks?
Week one: run the bus-factor audit on last month's escalations and rank the single-owner scenarios by recurrence. Ask your two most recent hires for their week-one question logs. Do not write anything yet.
Week two: write the top three runbooks in the format above, with explicit stop conditions and a stated expected output. Build one decision table for your most frequent judgment call, with real numbers in it. Then set the review rule, quarterly, claim it or it gets archived, and put the first review in the calendar before enthusiasm fades.
After that, add the second-occurrence rule to your team norms and let the library grow from real work rather than from a project plan. If you want the inbox, the articles and the handoff living in the same place while you do it, you can start a free trial and move the first three runbooks across this week.