A system can run perfectly and still become wrong

Lesson 07.1

Three kinds of health

A system can be operationally healthy while becoming less trustworthy. Nothing crashes. The AI still responds. The answers simply become less reliable.

Knowledge observability belongs inside AI observability. Tracking uptime alone will not tell you whether the system is using obsolete or contradictory information.

• Technical health — did the system run? Watch outages, latency, tool errors, failed and duplicated runs.
• Behavioural health — did it do the right thing? Watch evaluation results, overrides, scope violations and correction patterns.
• Knowledge health — was the evidence trustworthy? Watch freshness, contradictions, source coverage, obsolete citations and retrieval usefulness.

Lesson 07.2

Measure knowledge in two different ways

Repository health asks whether the underlying content is maintained: ownership, currency, duplication, approved-source coverage, conflicts and review compliance.

In-use usefulness asks whether that content answers real questions: retrieval relevance, context completeness, supported-answer rate, correct application and user feedback.

These measures are complementary. A tidy repository can fail its users. A popular AI experience can rely on dangerously stale sources.

Lesson 07.3

The operating loop

Observe → classify → assign → fix → test → release → monitor.

For each failure, identify whether it belongs to content, retrieval, instructions, tooling or permissions. Assign an accountable owner and track the result after the fix.

Separate reactive incident work from proactive maintenance. Regularly inspect frequently retrieved sources, repeated answer corrections and high-impact content nearing its review date.

• Weekly: inspect a representative sample of failed or corrected answers.
• Weekly: review high-use sources and unresolved content conflicts.
• After changes: run regression cases before release.
• Monthly: examine coverage, staleness, ownership and emerging failure patterns.
• At every handover: verify another person can explain, maintain and safely disable the system.