Skip to content

Commit 4b6d41b

Browse files
docs: DESIGN_PRINCIPLES.md — no conclusion may outrank its evidence
Eight principles, each extracted from a defect that shipped and each mapped back to it in a table, so none of them reads as taste. The one that generalises furthest is distinguishing 'not applicable' from 'not satisfied'. A rule that does not govern a case must never enter the evaluable set, because its unevaluated state will eventually be read as a verdict by something downstream — and the further that reading travels from the point of measurement, the more authoritative it looks. That failure lives in plumbing rather than logic: the correct answer is computed, then discarded by a layer that only knows two states. Also states the asymmetry between failing open and failing closed. Removing something the evidence does not condemn destroys information; asserting something it does not support fabricates it. A guard that deletes on uncertainty and one that asserts on uncertainty are different tools with the same name, and the direction must be chosen against the failure the operation can actually cause.
1 parent 2ba1cba commit 4b6d41b

2 files changed

Lines changed: 162 additions & 0 deletions

File tree

DESIGN_PRINCIPLES.md

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# Design principles
2+
3+
> **No conclusion may carry more certainty than the evidence it came from.**
4+
5+
Everything below follows from that one line. It is not a style preference: each
6+
principle here was extracted from a defect that shipped, and each defect was
7+
silent — the tool reported success while producing a wrong answer.
8+
9+
---
10+
11+
## 1. Never consume your own output as input
12+
13+
A tool that reads what it wrote will confirm itself.
14+
15+
0.1.0 matched over the whole markdown file, frontmatter and generated block
16+
included. It linked both example notes to `severity` — a function inside its own
17+
`split.py` — and reported *"2/2 notes linked"*. Every statistic downstream was
18+
formally correct and substantively false: the backlink table, the
19+
most-referenced ranking, the percentage of notes linked. All computed correctly,
20+
over evidence the tool had manufactured.
21+
22+
The same defect made links immortal. A symbol removed from the prose survived,
23+
because the next run rediscovered it in the block written by the last one.
24+
25+
**Rule.** Separate authored content from generated content with explicit
26+
markers, and strip the generated part before any analysis. If you cannot tell
27+
the two apart in code, you cannot tell them apart in a result either.
28+
29+
## 2. The documented contract must traverse the real path
30+
31+
0.2.0 documented `### STATUS: CLOSED` and did not honour it. The classifier
32+
understood the syntax perfectly; the caller filtered those headings out before
33+
calling it. A note marked closed came out open.
34+
35+
The tests missed it because they called the classifier directly, bypassing the
36+
very filter that broke it. **A test that skips the caller cannot see the
37+
caller's mistake.**
38+
39+
**Rule.** Test the documented entry point, not the function you happen to have
40+
written. A unit test that never touches the path a user takes is evidence about
41+
a function, not about a product.
42+
43+
## 3. Ambiguity is a state, not a tie to be broken
44+
45+
0.2.x kept one location per symbol name and discarded the rest. A note naming
46+
`validate`, where two modules define it, was linked to whichever definition the
47+
backend returned first — an answer that depended on filesystem order,
48+
reproducible only by coincidence, and carried no warning.
49+
50+
Collapsing several candidates into one is not a simplification. It is a claim of
51+
uniqueness that the evidence does not support.
52+
53+
**Rule.** Model the three outcomes distinctly:
54+
55+
```
56+
MATCH exactly one candidate, or the author disambiguated
57+
NO MATCH no candidate
58+
AMBIGUOUS several candidates and nothing chooses between them
59+
```
60+
61+
`AMBIGUOUS` is a result, and it is reported with all its candidates. Anything
62+
built on top can then reason about certainty instead of re-deriving it from an
63+
answer that looks definite.
64+
65+
## 4. Disambiguation is the author's decision, ranked by directness
66+
67+
```
68+
explicit override in the note's frontmatter
69+
> path named in the note
70+
> qualified name used in the note
71+
> nothing — stay ambiguous
72+
```
73+
74+
Each level is a statement the author made. None is an inference the tool made on
75+
their behalf.
76+
77+
**Rule.** Probabilistic resolution does not belong in the core. A future
78+
`suggest` command may propose candidates with a confidence and the evidence
79+
behind it — and must not turn a suggestion into a link until a human accepts it.
80+
The moment a guess is written into the artefact, it becomes indistinguishable
81+
from a fact.
82+
83+
## 5. Fail open when deleting, fail closed when asserting
84+
85+
The two directions are not symmetric, and the asymmetry follows from what is at
86+
stake.
87+
88+
- **Removing** something the evidence does not clearly condemn destroys
89+
information. When a symbol cannot be located, or a concept cannot be judged,
90+
keep it.
91+
- **Asserting** something the evidence does not clearly support fabricates
92+
information. When a criterion cannot be evaluated, say so; do not credit it.
93+
94+
**Rule.** Decide which failure your operation can cause, and default against it.
95+
A guard that deletes on uncertainty and a guard that asserts on uncertainty are
96+
different tools wearing the same name.
97+
98+
## 6. Distinguish "not applicable" from "not satisfied"
99+
100+
These are different answers and must not collapse into each other:
101+
102+
```
103+
present the evidence is there and supports it
104+
absent the evidence is there and contradicts it
105+
unknown the evidence is missing
106+
ambiguous the evidence supports several answers
107+
not applicable the rule does not govern this case at all
108+
```
109+
110+
A rule that does not apply should never enter the evaluable set. If it does, its
111+
"unevaluated" state will eventually be read as a verdict by something downstream
112+
— and the further that reading travels from the point of measurement, the more
113+
authoritative it looks.
114+
115+
This is the principle that generalises furthest, and it is the one most often
116+
lost in plumbing rather than in logic: the correct answer is computed, then
117+
discarded by a layer that only knows about two states.
118+
119+
## 7. Provenance and staleness are part of the result
120+
121+
An index says where things were, at a moment, according to some backend. Record
122+
which backend, which commit, and whether the scan was truncated. An answer
123+
derived from a stale index is not wrong — it is unfalsifiable, which is worse,
124+
because nothing about it looks doubtful.
125+
126+
**Rule.** Ship the metadata that lets a consumer distrust the result.
127+
128+
## 8. Report what you did not do
129+
130+
Counts that silently omit are the most persuasive form of wrong. If a run
131+
skipped files, truncated a scan, capped a list or could not judge an item, say
132+
so beside the number.
133+
134+
`282 of 300 replayed, 18 had no ingestion to replay` is a true statement.
135+
`282 replayed` is a misleading one, and it is misleading in the direction that
136+
flatters the tool.
137+
138+
---
139+
140+
## Where this came from
141+
142+
These are not theoretical. Each maps to a shipped defect, found by review or by
143+
measurement rather than by use:
144+
145+
| principle | defect |
146+
|---|---|
147+
| 1 | 0.1.0 linked notes to a symbol in its own source and called it success |
148+
| 2 | 0.2.0 documented a grammar the CLI never reached |
149+
| 3 | 0.2.x resolved duplicate names by filesystem order |
150+
| 5 | a guard deleted a condition asserted later in the same document |
151+
| 6 | a criterion that could not apply was credited as satisfied |
152+
| 8 | a completion figure that counted only what was attempted |
153+
154+
The pattern across all of them is the same, and it is worth naming: **the number
155+
was large and correct, and the phenomenon it described did not exist.** Silent
156+
failure is not the absence of output. It is confident output over evidence that
157+
was never checked.

README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,11 @@ when someone else's project changes shape.
149149

150150
## Design decisions worth knowing
151151

152+
They all follow from one line, set out in [DESIGN_PRINCIPLES.md](DESIGN_PRINCIPLES.md):
153+
154+
> **No conclusion may carry more certainty than the evidence it came from.**
155+
156+
152157
**tracelink never reads its own output.** Frontmatter and the managed block, delimited by
153158
`<!-- tracelink:linked-code:start -->` and `:end`, are stripped before matching. Without
154159
this the tool matches itself: 0.1.0's demo linked both example notes to

0 commit comments

Comments
 (0)