Skip to content

test: pin graph.json invariance across checkout folder, rebuild and line endings - #4218

Closed
rohit-jsfreaky wants to merge 1 commit into
Graphify-Labs:v8from
rohit-jsfreaky:test/graph-invariance
Closed

rohit-jsfreaky wants to merge 1 commit into
Graphify-Labs:v8from
rohit-jsfreaky:test/graph-invariance

Conversation

@rohit-jsfreaky

Copy link
Copy Markdown
Contributor

What does this PR do?

Closes #4217.

Adds tests/test_graph_invariance.py. It builds the whole tests/fixtures corpus (every extractor) through the real CLI (python -m graphify update <dir> --force, with and without --no-cluster) and checks the invariants that #4152, #4154, #4156, #4164, #4180, #4182 and #4188 each broke for one extractor or write path:

test catches
test_checkout_folder_does_not_change_the_graph checkout path inside an id (#4152, #4154, #4156)
test_graph_never_contains_the_checkout_path path in ids, edges or top-level keys (#4182, #4188)
test_rebuilding_an_unchanged_tree_changes_nothing first build != rebuild (#4180, #4188)
test_line_endings_do_not_change_the_graph CRLF ids != LF ids (#4164)
test_every_edge_endpoint_is_a_declared_node raw edges to undeclared nodes (#4182)
test_fixture_corpus_is_not_trivial a silent empty build passing everything above

Each runs for no_cluster and clustered (12 cases). Builds are shared through a module fixture, so the whole file takes about 20 s. Community ids are excluded from the comparison; clustering determinism is #3761's job.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Tests or CI
  • Refactor
  • Security fix

Verification & Invariants

Invariant: the same source tree gives the same graph.json whatever folder it is in, however many times it is rebuilt, and whatever its line endings; and the raw graph never references a node it does not declare.
Persisted state it could invalidate: none (test only, builds in pytest tmp dirs, leaves the repo clean).

Proof that it catches the real bugs:

graphify result
0.9.79 (before #4181, #4183, #4189) 3 failed, 9 passed: path in graph.json, rebuild != build, undeclared endpoints (all no_cluster)
0.9.80 (6478eb7) 12 passed, ~20 s
  • Read the CONTRIBUTING.md guide.
  • Reproduced the issue and identified the invariant.
  • Made the smallest fix necessary (one test file).
  • Added a regression test: it fails on 0.9.79 for exactly the three bugs fixed in 0.9.80.
  • Kept the PR description synchronized with the final implementation.
  • Documented any limitations / unsupported cases explicitly.

Limitations: community ids are not compared (see #3761). Binary and non-UTF-8 fixture files are copied as-is in the CRLF copy. The corpus is tests/fixtures, so a language without a fixture is not covered until it gets one.

How was this tested?

Project venv (Python 3.12.12), Windows 11, on v8 @ 6478eb7:

python -m pytest tests/test_graph_invariance.py -q        # 12 passed in ~20 s
python -m pytest tests -q                                  # 18 failed, 6775 passed, 27 skipped; all 18 are in the clean-v8 failure list on this machine (Windows), none in the new file, 0 new
python -m ruff check tests/test_graph_invariance.py        # All checks passed
pyright tests/test_graph_invariance.py                     # 0 errors

Graphify-specific checklist

  • I updated generated skill artifacts — not needed, no skill fragments touched.
  • I confirmed that AST/structural extraction remains deterministic (this test is that check).
  • I reviewed changes for security implications (subprocess args are a fixed list, no shell, no interpolation).
  • I confirmed that no API keys or local-only graph data are included.
  • I disclosed AI authorship in my commit messages.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B8gwEpjajiWHVW3c9smuKf

…ine endings

Seven bugs of one shape (Graphify-Labs#4152, Graphify-Labs#4154, Graphify-Labs#4156, Graphify-Labs#4164, Graphify-Labs#4180, Graphify-Labs#4182, Graphify-Labs#4188) were each
found by diffing two builds of the same tree by hand: a checkout path leaking into an
id, a rebuild writing a different graph than the first build, CRLF ids differing from
LF, an edge to a node that is never declared. Each fix got its own narrow test, so the
next extractor or write-path change can bring the same class back unseen.

tests/test_graph_invariance.py builds the whole tests/fixtures corpus (every extractor)
through the real CLI, --no-cluster and clustered, and checks: two folders give the same
graph, the checkout path appears nowhere in graph.json, a rebuild of the unchanged tree
changes nothing, LF and CRLF give the same graph, and every raw edge endpoint is declared.
On 0.9.79 the three checks those fixes cover fail; on 0.9.80 all 12 pass in about 20 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B8gwEpjajiWHVW3c9smuKf
@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown

Thanks for the pull request, @rohit-jsfreaky. A maintainer will review it soon.

Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions.

A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).


Graphify review — findings

Adds an invariance suite that builds the whole tests/fixtures corpus through the real graphify update CLI, both clustered and --no-cluster. It asserts that graph.json's nodes and edges come out identical across checkouts in different folders, a cold build versus a rebuild, and LF versus CRLF copies, with community/community_name excluded from the comparison. It also checks that no checkout path leaks into the output, that every edge endpoint is a declared node, and that the corpus yields over 500 nodes and edges so an empty build can't pass vacuously.

No blocking issues surfaced. 3 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 16 functions depend on the 16 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 16 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 16 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

1 of 345 test file(s) selected (0%) via static blast radius.

  • tests/test_graph_invariance.py — impact, changed-test

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Docs that may be stale (advisory)

…and 10 more.

@safishamsi

Copy link
Copy Markdown
Member

Landed in v0.9.81 via an authorship-preserving cherry-pick, so your commit is on v8 with you credited as the author. Closing as shipped — thanks @rohit-jsfreaky!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: one test that pins graph.json invariance for every extractor (checkout folder, rebuild, line endings)

2 participants