|
| 1 | +# Design principles |
| 2 | + |
| 3 | +> **No conclusion may carry more certainty than the evidence it came from.** |
| 4 | +
|
| 5 | +Everything below follows from that one line. It is not a style preference: each |
| 6 | +principle here was extracted from a defect that shipped, and each defect was |
| 7 | +silent — the tool reported success while producing a wrong answer. |
| 8 | + |
| 9 | +--- |
| 10 | + |
| 11 | +## 1. Never consume your own output as input |
| 12 | + |
| 13 | +A tool that reads what it wrote will confirm itself. |
| 14 | + |
| 15 | +0.1.0 matched over the whole markdown file, frontmatter and generated block |
| 16 | +included. It linked both example notes to `severity` — a function inside its own |
| 17 | +`split.py` — and reported *"2/2 notes linked"*. Every statistic downstream was |
| 18 | +formally correct and substantively false: the backlink table, the |
| 19 | +most-referenced ranking, the percentage of notes linked. All computed correctly, |
| 20 | +over evidence the tool had manufactured. |
| 21 | + |
| 22 | +The same defect made links immortal. A symbol removed from the prose survived, |
| 23 | +because the next run rediscovered it in the block written by the last one. |
| 24 | + |
| 25 | +**Rule.** Separate authored content from generated content with explicit |
| 26 | +markers, and strip the generated part before any analysis. If you cannot tell |
| 27 | +the two apart in code, you cannot tell them apart in a result either. |
| 28 | + |
| 29 | +## 2. The documented contract must traverse the real path |
| 30 | + |
| 31 | +0.2.0 documented `### STATUS: CLOSED` and did not honour it. The classifier |
| 32 | +understood the syntax perfectly; the caller filtered those headings out before |
| 33 | +calling it. A note marked closed came out open. |
| 34 | + |
| 35 | +The tests missed it because they called the classifier directly, bypassing the |
| 36 | +very filter that broke it. **A test that skips the caller cannot see the |
| 37 | +caller's mistake.** |
| 38 | + |
| 39 | +**Rule.** Test the documented entry point, not the function you happen to have |
| 40 | +written. A unit test that never touches the path a user takes is evidence about |
| 41 | +a function, not about a product. |
| 42 | + |
| 43 | +## 3. Ambiguity is a state, not a tie to be broken |
| 44 | + |
| 45 | +0.2.x kept one location per symbol name and discarded the rest. A note naming |
| 46 | +`validate`, where two modules define it, was linked to whichever definition the |
| 47 | +backend returned first — an answer that depended on filesystem order, |
| 48 | +reproducible only by coincidence, and carried no warning. |
| 49 | + |
| 50 | +Collapsing several candidates into one is not a simplification. It is a claim of |
| 51 | +uniqueness that the evidence does not support. |
| 52 | + |
| 53 | +**Rule.** Model the three outcomes distinctly: |
| 54 | + |
| 55 | +``` |
| 56 | +MATCH exactly one candidate, or the author disambiguated |
| 57 | +NO MATCH no candidate |
| 58 | +AMBIGUOUS several candidates and nothing chooses between them |
| 59 | +``` |
| 60 | + |
| 61 | +`AMBIGUOUS` is a result, and it is reported with all its candidates. Anything |
| 62 | +built on top can then reason about certainty instead of re-deriving it from an |
| 63 | +answer that looks definite. |
| 64 | + |
| 65 | +## 4. Disambiguation is the author's decision, ranked by directness |
| 66 | + |
| 67 | +``` |
| 68 | +explicit override in the note's frontmatter |
| 69 | +> path named in the note |
| 70 | +> qualified name used in the note |
| 71 | +> nothing — stay ambiguous |
| 72 | +``` |
| 73 | + |
| 74 | +Each level is a statement the author made. None is an inference the tool made on |
| 75 | +their behalf. |
| 76 | + |
| 77 | +**Rule.** Probabilistic resolution does not belong in the core. A future |
| 78 | +`suggest` command may propose candidates with a confidence and the evidence |
| 79 | +behind it — and must not turn a suggestion into a link until a human accepts it. |
| 80 | +The moment a guess is written into the artefact, it becomes indistinguishable |
| 81 | +from a fact. |
| 82 | + |
| 83 | +## 5. Fail open when deleting, fail closed when asserting |
| 84 | + |
| 85 | +The two directions are not symmetric, and the asymmetry follows from what is at |
| 86 | +stake. |
| 87 | + |
| 88 | +- **Removing** something the evidence does not clearly condemn destroys |
| 89 | + information. When a symbol cannot be located, or a concept cannot be judged, |
| 90 | + keep it. |
| 91 | +- **Asserting** something the evidence does not clearly support fabricates |
| 92 | + information. When a criterion cannot be evaluated, say so; do not credit it. |
| 93 | + |
| 94 | +**Rule.** Decide which failure your operation can cause, and default against it. |
| 95 | +A guard that deletes on uncertainty and a guard that asserts on uncertainty are |
| 96 | +different tools wearing the same name. |
| 97 | + |
| 98 | +## 6. Distinguish "not applicable" from "not satisfied" |
| 99 | + |
| 100 | +These are different answers and must not collapse into each other: |
| 101 | + |
| 102 | +``` |
| 103 | +present the evidence is there and supports it |
| 104 | +absent the evidence is there and contradicts it |
| 105 | +unknown the evidence is missing |
| 106 | +ambiguous the evidence supports several answers |
| 107 | +not applicable the rule does not govern this case at all |
| 108 | +``` |
| 109 | + |
| 110 | +A rule that does not apply should never enter the evaluable set. If it does, its |
| 111 | +"unevaluated" state will eventually be read as a verdict by something downstream |
| 112 | +— and the further that reading travels from the point of measurement, the more |
| 113 | +authoritative it looks. |
| 114 | + |
| 115 | +This is the principle that generalises furthest, and it is the one most often |
| 116 | +lost in plumbing rather than in logic: the correct answer is computed, then |
| 117 | +discarded by a layer that only knows about two states. |
| 118 | + |
| 119 | +## 7. Provenance and staleness are part of the result |
| 120 | + |
| 121 | +An index says where things were, at a moment, according to some backend. Record |
| 122 | +which backend, which commit, and whether the scan was truncated. An answer |
| 123 | +derived from a stale index is not wrong — it is unfalsifiable, which is worse, |
| 124 | +because nothing about it looks doubtful. |
| 125 | + |
| 126 | +**Rule.** Ship the metadata that lets a consumer distrust the result. |
| 127 | + |
| 128 | +## 8. Report what you did not do |
| 129 | + |
| 130 | +Counts that silently omit are the most persuasive form of wrong. If a run |
| 131 | +skipped files, truncated a scan, capped a list or could not judge an item, say |
| 132 | +so beside the number. |
| 133 | + |
| 134 | +`282 of 300 replayed, 18 had no ingestion to replay` is a true statement. |
| 135 | +`282 replayed` is a misleading one, and it is misleading in the direction that |
| 136 | +flatters the tool. |
| 137 | + |
| 138 | +--- |
| 139 | + |
| 140 | +## Where this came from |
| 141 | + |
| 142 | +These are not theoretical. Each maps to a shipped defect, found by review or by |
| 143 | +measurement rather than by use: |
| 144 | + |
| 145 | +| principle | defect | |
| 146 | +|---|---| |
| 147 | +| 1 | 0.1.0 linked notes to a symbol in its own source and called it success | |
| 148 | +| 2 | 0.2.0 documented a grammar the CLI never reached | |
| 149 | +| 3 | 0.2.x resolved duplicate names by filesystem order | |
| 150 | +| 5 | a guard deleted a condition asserted later in the same document | |
| 151 | +| 6 | a criterion that could not apply was credited as satisfied | |
| 152 | +| 8 | a completion figure that counted only what was attempted | |
| 153 | + |
| 154 | +The pattern across all of them is the same, and it is worth naming: **the number |
| 155 | +was large and correct, and the phenomenon it described did not exist.** Silent |
| 156 | +failure is not the absence of output. It is confident output over evidence that |
| 157 | +was never checked. |
0 commit comments