Skip to content

Fix audit findings: templates, gates, CI supply chain - #3

Merged
RandyNorthrup merged 6 commits into
mainfrom
audit-hardening
Sep 24, 2026
Merged

RandyNorthrup merged 6 commits into
mainfrom
audit-hardening

Conversation

@RandyNorthrup

Copy link
Copy Markdown
Owner

Summary

Fixes from a full audit of v0.6.0: shipped template defects, gates this package prescribes but never ran on itself, and CI supply-chain gaps. Every finding was reproduced before it was fixed, and every changed rule was red-drilled (green → intended red → byte-identical restore → green). Evidence is in docs/QUALITY-REVIEW.md → "Post-release audit: 2026-09-24".

Shipped defects fixed

  • templates/csharp/Directory.Build.props has been malformed XML since v0.5.0 (-- inside a comment → MSB4024, the file cannot load).
  • The PSScriptAnalyzer template could pass while losing findings: PSUseCorrectCasing intermittently throws. A planted violation went unreported with exit 0. The rule is off, and every documented command adds -ErrorAction Stop.
  • The C# dead-code gate (IDE0051/0052) never fired: no severities were configured. Added the .editorconfig template.
  • Pre-commit template:
    • The vulture hook exited 2 because no paths were configured.
    • The mypy hook ran in an isolated environment where dependency types became Any.
    • The ShellCheck hook required Docker.
    • mixed-line-ending --fix=lf rewrote CRLF-by-policy files.
  • verify-format-safe.py reported unreadable input as "AST changed"; the plan writer left PLAN.md at mode 0600 on POSIX.
  • detect-stack.sh: its JSON root was unescaped, and it matched case-sensitively, unlike the PowerShell scanner. It is now one pass instead of 20 (480 s → 16 s on a large tree).

Added: a Rust deny.toml, five stricter mypy codes, actionlint/zizmor/shebang hooks, broader scanner detection, and guidance for test depth, performance, supply chain, and CI hardening.

Repository gates

  • A frozen pre-commit config runs Ruff, both mypy scopes (shipped template), vulture, Bandit, ShellCheck, shfmt, PSScriptAnalyzer, actionlint, and zizmor.
  • CI adds a checksum-verified full-history gitleaks scan, a coverage floor (82%), and hash-locked tool installs.
  • Every action is pinned to a commit SHA, with persist-credentials: false and job timeouts.
  • Dependabot has a 7-day cooldown.

Test plan

  • 58 unit tests on Windows (Python 3.14, 3.12) and Linux (3.14); coverage 83.1% / 83.3% ≥ 82%
  • pre-commit run on all files: 26 hooks pass
  • cross-platform-smoke.ps1 -RedDrills: 6 drills pass on PowerShell 7 and 5.1
  • release-package-smoke.ps1 passes on PowerShell 7 and 5.1 (committed tree)
  • Template drills: mypy ×5, pre-commit hooks ×5, C# IDE rules ×5, cargo-deny ×2, PSScriptAnalyzer stability
  • Hosted matrix on this PR (macOS and POSIX parity/escaping are first exercised here)

🤖 Generated with Claude Code

RandyNorthrup and others added 6 commits September 24, 2026 16:35
The C# Directory.Build.props template has been malformed XML since
v0.5.0: an XML comment contained "--", so MSBuild refused to load it.
PSUseCorrectCasing intermittently crashed PSScriptAnalyzer and dropped
its findings while -EnableExit still exited 0; the rule is off and
every documented command adds -ErrorAction Stop.

The pre-commit template's vulture hook failed without paths, mypy ran
where dependency types were Any, ShellCheck needed Docker, and
mixed-line-ending --fix=lf rewrote CRLF-by-policy files. Add the C#
dead-code severities and a cargo-deny policy, stricter mypy codes,
shebang and workflow hooks, and a test that parses every template.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
verify-format-safe.py let missing, undecodable, or directory input
escape as an uncaught exception, which exits 1: the code its contract
reserves for a changed AST. It also decoded source as UTF-8 instead of
honoring PEP 263 declarations. It now parses bytes and exits 2 for any
input it cannot compare.

The atomic plan writer replaced PLAN.md with its owner-only temporary
file, so the plan became mode 0600 on POSIX. Preserve the existing
mode, or use the umask for a new plan, and flush the directory entry
after the rename.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The package asked projects to run ShellCheck, shfmt, PSScriptAnalyzer,
vulture, Bandit, and coverage but ran none of them on itself, and CI
type-checked only the delivery package, not the tests. Add a frozen
pre-commit configuration with those hooks, both mypy scopes under the
shipped template, and a coverage floor that counts child processes.

Tests now pass the template: JSON fixtures use the documented Any
escape hatch, setUp overrides use @OverRide, and trusted subprocess
calls carry reviewed Bandit exceptions. A type-checked whitelist
covers names vulture cannot trace. Tools install from a universal,
hash-locked requirements file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The release job holds contents, attestation, and OIDC write access but
used actions by movable tag and left the checkout token in .git/config.
Pin every action to its commit SHA, disable persisted credentials, and
set job timeouts.

Add a lint job that runs the pre-commit gates and a checksum-verified
full-history gitleaks scan. The Python matrix installs hash-locked
tools, type-checks scripts and tests with the shipped template, and
enforces the coverage floor. The POSIX parity job also checks JSON
escaping of an awkward directory name. Dependabot proposes updates for
actions, hooks, and tools after a seven-day cooldown.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
detect-stack.sh wrote the root path into JSON unescaped, matched
extensions and pruned directories case-sensitively unlike the
PowerShell scanner, and walked the tree once per extension: 480 s
versus 16 s on a large workspace. It now makes one escaped,
case-insensitive pass and passes ShellCheck with every check enabled.

Both scanners now report knip.jsonc (the package's own recommended
name), Go and golangci-lint configuration, update-bot configuration,
lockfiles, toolchain pins, and dpdm, Go tools, bats, actionlint,
zizmor, OSV-Scanner, hadolint, PSScriptAnalyzer, and Pester 5. The
smoke suite asserts them and keeps six red drills, including
case-insensitive pruning and SHA-pinned release actions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add source-backed guidance for coverage floors, mutation testing,
property and fuzz testing, benchmarks, dependency supply chain, and
hardened GitHub Actions workflows, plus axe-core and React hook rules.
Both setup and retrofit workflows now require locked installs, pinned
CI actions, and a measured coverage floor.

Record every reproduced finding, red drill, environment, and open gate
in the dated quality review, the plan, and the changelog. Update the
contributor workflow for the hash-locked tools and pre-commit hooks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-24T23:41:48.132253Z 1c43b29 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@RandyNorthrup
RandyNorthrup merged commit 1c43b29 into main Sep 24, 2026
13 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1c43b296a3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


```bash
CACHES=(.ruff_cache .mypy_cache .pytest_cache .tox htmlcov)
CACHES=(.ruff_cache .mypy_cache .pytest_cache .tox htmlcov .coverage)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Track parallel coverage data files during cleanup

When the target project enables coverage parallel mode—explicitly, or via coverage.py's subprocess patch—the baseline creates .coverage.* files rather than only .coverage. Both cleanup snippets and the later ignore guidance list only the exact .coverage name, so these files remain untracked and violate this phase's promised clean-worktree invariant. Snapshot and remove newly created .coverage.* files while preserving pre-existing ones, and include the pattern in the phase-2 ignore guidance.

Useful? React with 👍 / 👎.

@RandyNorthrup
RandyNorthrup deleted the audit-hardening branch September 24, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant