Merge pull request #152 from ActiveMemory/design/pd-m4-conventions

pd-m4: teach progressive disclosure the convention kind

Author: josealekhine

.context/CONVENTIONS.md

@@ -45,6 +45,20 @@ For significant decisions:
 
 -->
 
+## [2026-07-19-100259] M4 conventions digestion: curated ## -section taxonomy, unified into the entry-kind mover
+
+**Status**: Accepted
+
+**Context**: CONVENTIONS.md is the last unbounded canonical root (390 lines, 18 stable curated ## sections). The M3 mover refuses the convention kind (ErrApplyNotEntryKind). The spec assumed conventions accrete as ### entries under a ## Recent staging zone, but the real file is a curated ## -section taxonomy with no ## Recent and no timestamps — the blocking TBD for M4.
+
+**Decision**: M4 conventions digestion: curated ## -section taxonomy, unified into the entry-kind mover
+
+**Rationale**: Approach A (unify) over a separate convention path or a one-time script: the file already fits the entry-kind preamble|staging|## Themes shape, so parametrizing the mover by a per-kind entry prefix (## for conventions vs ## [) removes special-case code instead of adding a parallel path. Identity = section title (no timestamp). Keep the scanner dumb — no fence detection (a rabbit hole in a destructive parser); byte-conservation plus the human-gated plan review make a dumb scanner safe against false ## boundaries. Curated-taxonomy matches what conventions are: stable, not accreting.
+
+**Consequence**: Retire parseConvention, ConventionLinePrefix and HeadingRecent (## Recent); the three-self-similar-tiers claim becomes literally true. New sentinel ErrDuplicateStagedTitle (fail-loud on duplicate ## titles). heading.ParseEntryBlocks stays untouched (zero regression to the live LEARNINGS/DECISIONS path). ctx convention add joins the unified prepend anchor (insert before the first ## section, skipping the structural ## Themes; fallback AfterHeader), landing new conventions newest-first in staging above ## Themes — correcting the old AppendAtEnd inconsistency (decisions/learnings already prepend). This add-path change is IN M4 scope. specs/progressive-disclosure.md revised accordingly.
+
+---
+
 ## Themes
 
 - package-structure-and-quality-gates — Package taxonomy & quality gates: write/ output, internal/err, config/ constants, doc.go floor, AST audit tests, log split, GraphBuilder interface → [package-structure-and-quality-gates](decisions/package-structure-and-quality-gates.md)

.context/DECISIONS.md

@@ -15,6 +15,26 @@ DO NOT UPDATE FOR:
 -->
 
 
+## [2026-07-19-210439] The disclosure parser is a deliberately dumb line-scanner (skips <!-- --> comments, not code fences)
+
+**Context**: M4 unifies conventions into the ## -section entry model, where ## collides far more than ## [. Fence detection was considered and rejected.
+
+**Lesson**: Fence detection (nested/tilde/indented fences) is a rabbit hole in a parser that does destructive moves. Safety comes from byte-conservation (a mis-cut is a reversible mis-grouping, not loss) plus the human-gated plan review (a false entry shows up at inspect).
+
+**Application**: Don't add fence-awareness to the disclosure scanner; a stray ## inside a fenced code block is fixed at authoring time (rewrite that one line), never by the tool.
+
+---
+
+## [2026-07-19-210439] Measurement gates surface a real bug in every disclosure milestone
+
+**Context**: pd-m1's add-path measurement gate flushed the insert.AfterHeader tail-truncation bug; pd-m3's T17 real rollout surfaced the DECISIONS-template comment-parse bug (a ## [ inside <!-- --> read as staging, so Validate returned ErrStagingUnparsable once the fold emptied real staging).
+
+**Lesson**: For the disclosure work, driving the REAL fold — not unit tests — is where bugs hide. The measurement gate is not ceremony.
+
+**Application**: Treat pd-m4 T20 (drive the digest on the real CONVENTIONS.md) as the authoritative gate; expect it to find something; verify conservation + all invariants on the real file before T21's human-gated rollout.
+
+---
+
 ---
 
 ## Themes

.context/LEARNINGS.md

@@ -2998,7 +2998,35 @@ not agent discipline — the clobber risk class the M1 guards exist for.
 
 - [x] [E2] `ctx disclosure apply` CLI — reads plan JSON, refuses non-knowledge files + convention kind, write-safe on error, doc.go + wiring guards (T10–T13). Plan: specs/plans/pd-m3.md #priority:medium #session:f706d9de #branch:design/progressive-disclosure #added:2026-07-18
 
-- [ ] [E3] `ctx-digest` apply-path skill + copilot sync; MEASUREMENT GATE T16: driven apply on a realistic fixture moves entries + writes gists losslessly; real LEARNINGS→DECISIONS rollout is human-gated (T17); milestone gate (T14–T18). Plan: specs/plans/pd-m3.md #priority:medium #session:f706d9de #branch:design/progressive-disclosure #added:2026-07-18
+- [x] [E3] `ctx-digest` apply-path skill + copilot sync; MEASUREMENT GATE T16: driven apply on a realistic fixture moves entries + writes gists losslessly; real LEARNINGS→DECISIONS rollout is human-gated (T17); milestone gate (T14–T18). Plan: specs/plans/pd-m3.md #priority:medium #session:f706d9de #branch:design/progressive-disclosure #added:2026-07-18
+
+### Progressive disclosure — Milestone 4 (CONVENTIONS)
+
+Plan: `specs/plans/pd-m4.md` · Spec: `specs/progressive-disclosure.md`
+Milestone 4 teaches the mover the **convention kind**: CONVENTIONS.md's 18
+curated `## ` sections fold into per-theme tier-1 files under the same
+`preamble | staging | ## Themes` layout as entry kinds (per-kind entry
+prefix `## ` vs `## [`, title identity). Retires the `### `-under-`##
+Recent` model; `ctx convention add` prepends above `## Themes`.
+
+**Completion rule**: an epic is `[x]` only when every task in its range is
+`[x]`/`[o]` in `specs/plans/pd-m4.md` (the plan is the source of truth).
+Epics partition T01–T22: E1[T01–04] E2[T05–09] E3[T10–12] E4[T13–15]
+E5[T16–17] E6[T18–19] E7[T20–22] = 22.
+
+- [x] [E1] Config + vocabulary: `EntryPrefix(Kind)`, retire `ConventionLinePrefix`/`HeadingRecent`, `ErrDuplicateStagedTitle`, `ThemeDir(convention)` (T01–T04). Plan: specs/plans/pd-m4.md #priority:medium #session:8bc7532d #branch:design/pd-m4-conventions #added:2026-07-19
+
+- [x] [E2] Parser unification: convention `## `-enumerator (title identity), prefix-parametrized `parseEntryKind`, delete `parseConvention`, kind-aware inspect, `SplitStaging` on `## ` (T05–T09). Plan: specs/plans/pd-m4.md #priority:medium #session:8bc7532d #branch:design/pd-m4-conventions #added:2026-07-19
+
+- [x] [E3] Validate: generalize entry-below-themes to per-kind prefix, dup-title fail-loud (`ErrDuplicateStagedTitle`), rule-3 via enumerator (T10–T12). Plan: specs/plans/pd-m4.md #priority:medium #session:8bc7532d #branch:design/pd-m4-conventions #added:2026-07-19
+
+- [x] [E4] Mover: lift `ErrApplyNotEntryKind` for conventions, Apply end-to-end + title-only identity through the plan (T13–T15). Plan: specs/plans/pd-m4.md #priority:medium #session:8bc7532d #branch:design/pd-m4-conventions #added:2026-07-19
+
+- [x] [E5] Add-path: `ctx convention add` prepends above `## Themes` (was AppendAtEnd) + post-fold invariant test (T16–T17). Plan: specs/plans/pd-m4.md #priority:medium #session:8bc7532d #branch:design/pd-m4-conventions #added:2026-07-19
+
+- [x] [E6] `ctx-digest` skill — convention path + copilot sync (T18–T19). Plan: specs/plans/pd-m4.md #priority:medium #session:8bc7532d #branch:design/pd-m4-conventions #added:2026-07-19
+
+- [x] [E7] MEASUREMENT GATE T20 (drive digest on realistic CONVENTIONS fixture); real CONVENTIONS rollout human-gated (T21); milestone gate (T20–T22). Plan: specs/plans/pd-m4.md #priority:medium #session:8bc7532d #branch:design/pd-m4-conventions #added:2026-07-19
 
 - [ ] Progressive disclosure for canonical context files: the growth warnings (LEARNINGS/DECISIONS/CONVENTIONS over threshold) are NOT redundancy — consolidation only got LEARNINGS 98→88 because the entries are distinct, dense signal. The real lever is a structural pass: canonical files carry a tight summary/index and detail loads on demand (via `ctx index`/`ctx search` projection + an archive/detail tier). Manual design exercise first (/ctx-brainstorm → spec), then codify the repeatable procedure as a new skill (e.g. /ctx-progressive-disclosure). This exercise IS the baseline for the skill. #priority:medium #session:87e465a0 #branch:main #added:2026-07-16
   DESIGN DONE 2026-07-16 (session 87e465a0): /ctx-brainstorm run to

.context/TASKS.md

@@ -0,0 +1,103 @@
+# cli
+
+## CLI Structure
+
+- **CLI package taxonomy**: Every package under `internal/cli/` follows:
+  parent.go (Cmd wiring), doc.go, `cmd/root/` or `cmd/<sub>/`
+  (implementation), `core/` (shared helpers)
+- **cmd/ directories**: Only cmd.go, run.go, and tests — helpers and
+  output go to `core/`
+- **core/ structs**: Consolidated into a single `types.go` file
+- **User-facing text via assets**: All text routed through
+  `internal/assets` with YAML-backed TextDescKeys — no inline strings
+  in `core/` or `cmd/` packages
+- **config/ doc.go**: Every package under `internal/config/` must have
+  a doc.go with the project header and a one-line package comment
+- **DescKey prefix**: Not CmdDescKey — `cmd.DescKeyFoo` not
+  `cmd.CmdDescKeyFoo` (Go package hygiene, avoids stutter)
+- **Cobra Use: fields**: Must reference `cmd.Use*` constants, never raw
+  strings or `cmd.DescKey*`
+- **Run functions exported PascalCase**: `Run`, `RunImport`,
+  `RunArchive` etc. No private `runXXX` variants
+- **write/ packages write to stdio only**: Functions take
+  `*cobra.Command`, not `io.Writer`. Exception: `write/rc` writes to
+  `os.Stderr` because rc loads before cobra
+- **Package directory names singular**: Unless Go convention requires
+  plural
+- **Import grouping**: stdlib — blank line — external deps (cobra,
+  yaml) — blank line — ctx imports. Three groups, always in this order
+- **camelCase import aliases**: `cFlag` not `cflag`, `cfgFmt` not
+  `cfgfmt`
+- **Icons and symbols as token constants**: Not unicode escapes
+- **Cross-cutting domain types in internal/entity**: Types used by one
+  package stay in that package; types used across packages go to entity
+
+- Warn format strings centralized in config/warn/ — use warn.Close,
+  warn.Write, warn.Remove, warn.Mkdir, warn.Rename, warn.Walk, warn.Getwd,
+  warn.Readdir, warn.Marshal instead of inline format strings in log.Warn calls
+
+- Nav frontmatter title: fields must not contain ctx — frontmatter does not
+  support backticks, so the brand stays out of nav titles entirely (Hub, not The
+  ctx Hub). Body headings can use `ctx` since markdown supports backticks.
+
+- CLI flags and slash-commands inside headings or admonition titles must be
+  backticked: `--keep-frontmatter=false`, `/ctx-reflect`. The title-case engine
+  in hack/title-case-headings.py protects these patterns automatically, but
+  authors should still backtick at write time for clarity.
+
+- File extensions inside headings must be backticked when title-case
+  capitalization would otherwise apply: write `CONSTITUTION.md`, not
+  CONSTITUTION.Md. The title-case engine refuses to capitalize lowercase tokens
+  following a literal . dot, but explicit backticks remain the clearest signal.
+- New editor integrations include an MCP-merge test covering: create / empty
+  file / preserve existing keys / skip when registered / reject malformed JSON
+
+- Substrate vs. artifact placement: cognitive substrate (consumed and mutated
+  via ctx-mediated paths — `ctx agent`, `ctx decision add`, `/ctx-kb-ingest`,
+  `/ctx-handover`, ceremonies) lives under `.context/`; project artifacts (read
+  and edited directly by humans — `specs/`, `CLAUDE.md`, `GETTING_STARTED.md`,
+  `docs/`) live at the project root; tool config and tool homes (`.ctxrc`,
+  `.claude/`) live at root by dotfile/tool convention. The kb is substrate, not
+  artifact: direct file edits remain possible per Invariant 1, but the
+  skill-mediated path is the discipline. Rationale recorded in DECISIONS.md.
+
+## User-Facing Surface Completeness
+
+When a change adds or alters a user-facing surface — a new
+`ctx` subcommand, a new flag, an observable behavior change,
+a new exit shape, a new output line — the work is **not
+complete** until every one of the following has been updated
+in the same commit (or the same stacked PR, with the user's
+explicit OK):
+
+- `internal/assets/commands/commands.yaml` and
+  `examples.yaml` for the subcommand description and example
+- `internal/assets/claude/skills/ctx-<area>/SKILL.md` so the
+  agent knows the surface exists and when to trigger it
+- `internal/assets/integrations/copilot-cli/skills/<...>` if
+  a parallel skill exists for the integration
+- `docs/recipes/<related-recipe>.md` for any recipe that
+  already demonstrates the broader feature; consider a new
+  recipe if the surface is its own workflow shape
+- `docs/cli/<command>.md` if a per-command CLI doc page
+  exists for this surface
+
+Splitting these into a "Phase 2 / follow-up commit / future
+sweep" is **deferral** in the Constitution's sense, no matter
+how the phase is labeled. Docs are part of the deliverable,
+not a separable improvement. The "I can create a follow-up
+task" prohibition applies verbatim.
+
+Acceptable exceptions (state them in the commit body):
+
+- The surface is internal-only (no human user encounters it).
+- A recipe / skill genuinely does not exist for this feature
+  area and writing one is itself a larger separable piece of
+  work (then file the spec for that piece in the same commit,
+  do not just defer).
+
+The Self-check before declaring a feature commit complete is:
+*"If a user runs `ctx help` or asks `/ctx-<area>` to do this
+new thing today, will the help text / skill / recipe match
+what the code does?"* If no, the commit is not complete.
+

.context/conventions/cli.md

@@ -0,0 +1,105 @@
+# code style
+
+## Naming
+
+- **Constants use semantic prefixes**: Group related constants with prefixes
+  - `Dir*` for directories (`DirContext`, `DirArchive`)
+  - `File*` for file paths (`FileSettings`, `FileClaudeMd`)
+  - `Filename*` for file names only (`FilenameTask`, `FilenameDecision`)
+  - `*Type*` for enum-like values (`UpdateTypeTask`, `UpdateTypeDecision`)
+- **Package name = folder name**: Go canonical pattern
+  - `package initialize` in `initialize/` folder
+  - Never `package initcmd` in `init/` folder
+- **Go package names: lowercase, no underscores, no
+  mixedCaps**: per the [Effective Go](https://go.dev/blog/package-names)
+  guidance and the stdlib precedent (`strconv`, `httptest`,
+  `bufio`). Apply to the directory too — `internal/flagbind/`,
+  not `internal/flag_bind/`. Filenames may use underscores
+  (`foo_test.go` is canonical); package names may not. When in
+  doubt, find the closest stdlib analogue and copy its shape.
+- **Maps reference constants**: Use constants as keys, not literals
+  - `map[string]X{ConstKey: value}` not `map[string]X{"literal": value}`
+
+## Casing
+
+- **Proper nouns keep their casing** in comments, strings, and docs
+  - `Markdown` not `markdown` (it's a language name)
+  - `YAML`, `JSON`, `TOML` — always uppercase
+  - `GitHub`, `JavaScript`, `PostgreSQL` — match official casing
+  - Exception: code fence language identifiers are lowercase (`` ```markdown ``)
+
+## Predicates
+
+- **No Is/Has/Can prefixes**: `Completed()` not
+  `IsCompleted()`, `Empty()` not `IsEmpty()`
+- Applies to all bool-returning funcs and methods, exported or
+  not (`topicNested`, not `hasNestedTopic`)
+
+## Line Width
+
+- **Target ~80 characters**: Highly encouraged, not a hard limit
+  - Some lines will naturally exceed it (long strings,
+    struct tags, URLs) — that's fine
+  - Drift accumulates silently, especially in test code
+  - Break at natural points: function arguments, struct fields, chained calls
+
+## Patterns
+
+- **Centralize magic strings**: All repeated literals
+  belong in a `config` or `constants` package
+  - If a string appears in 3+ files, it needs a constant
+  - If a string is used for comparison, it needs a constant
+- **Path construction**: Always use stdlib path joining
+  - Go: `filepath.Join(dir, file)`
+  - Python: `os.path.join(dir, file)`
+  - Node: `path.join(dir, file)`
+  - Never: `dir + "/" + file`
+- **Constants reference constants**: Self-referential definitions
+  - `FileType[UpdateTypeTask] = FilenameTask` not
+    `FileType["task"] = "TASKS.md"`
+- **No error variable shadowing**: Use descriptive names
+  when multiple errors exist in a function
+  - `readErr`, `writeErr`, `indexErr` — not repeated `err` / `err :=`
+  - Shadowed `err` silently disconnects from the outer
+    variable, causing subtle bugs
+- **Colocate related code**: Group by feature, not by type
+  - `session/run.go`, `session/types.go`, `session/parse.go`
+  - Not: `runners/session.go`, `types/session.go`, `parsers/session.go`
+
+## Duplication
+
+- **Non-test code**: Apply the rule of three — extract
+  when a block appears 3+ times
+  - Watch for copy-paste during task-focused sessions
+    where the agent prioritizes completion over shape
+- **Test code**: Some duplication is acceptable for readability
+  - When the same setup/assertion block appears 3+ times, extract a test helper
+  - Use `t.Helper()` so failure messages point to the caller, not the helper
+
+## Error Handling
+
+- **Zero silent error discard**: Handle every error, never suppress with
+  `_ =` or `//nolint:errcheck`. Production: defer-close logs to stderr
+  via `log.Warn()`. Test: `t.Fatal(err)` for setup, `t.Log(err)` for
+  cleanup. For gosec false positives: fix the code rather than adding
+  nolint markers — the goal is zero golangci-lint suppressions
+- **Error constructors in internal/err**: Never in per-package err.go
+  files — eliminates the broken-window pattern where agents add local
+  errors when they see a local err.go exists
+- **Identity sentinels are `entity.Sentinel` consts, not
+  `errors.New`**: Declare `errors.Is` targets as
+  `const ErrX = entity.Sentinel(text.DescKey...)`. The
+  user-facing text lives in `commands/text/errors.yaml` keyed by
+  `err.<pkg>.<name>`; the sentinel's `Error()` resolves it via
+  `desc.Text` at call time. Never write
+  `var ErrX = errors.New("english")` — the English leaks into
+  `.Error()` output and bypasses localization. Never add an
+  `ErrMsg* = "english"` const layer in `internal/config/<pkg>/`
+  to back the sentinel; that layer is dead text once the typed
+  Sentinel does the lookup itself.
+- **Parameterised errors use typed structs**: When the error
+  needs to carry fields (path, name, etc.), define a struct in
+  `internal/err/<area>/` with a pointer-receiver `Error()` and
+  optional `Is(error) bool` for sentinel-compatibility. See
+  `internal/err/context.NotFoundError` for the canonical shape.
+