Skip to content

fix(compilers/openapi): resolve aliases and merge keys in the cycle pre-scan - #94

Merged
OmarAlJarrah merged 3 commits into
mainfrom
fix/openapi-cycle-scan-alias-and-contentschema
Jul 27, 2026
Merged

fix(compilers/openapi): resolve aliases and merge keys in the cycle pre-scan#94
OmarAlJarrah merged 3 commits into
mainfrom
fix/openapi-cycle-scan-alias-and-contentschema

Conversation

@OmarAlJarrah

@OmarAlJarrah OmarAlJarrah commented Jul 27, 2026

Copy link
Copy Markdown
Member

Summary

The pre-parse cycle scan reads the raw yaml.Node tree, while speakeasy's unmarshaller reads a
resolved one — aliases (keys and values alike) dereferenced, << merge keys expanded. Wherever
the scan's node model was narrower than that, a genuine $ref cycle slipped past it and reached the
reference resolver, which dies with fatal error: stack overflow. That fault is unrecoverable, so
the process crashes instead of emitting the openapi/cyclic-ref diagnostic the scan exists to
produce.

Six inputs reproduced the crash on main. All are parseable documents:

Shape Why the scan missed it
$ref whose value is a YAML alias pureRefTarget required the value node to be a scalar
$ref under contentSchema contentSchema was absent from the sub-schema key set
$ref key that is itself an alias key comparison read the alias node's raw Value
$ref pulled in by a << merge key the mapping carries no literal $ref key at all
schema node that is an alias the schema walk bailed on Kind != MappingNode
one anchored $ref node reused in two schema positions surfaced by the memoization below

The change

A single resolver-faithful view of the tree. Every read of a mapping in the scan now goes
through nodeView, which returns a mapping's effective pairs the way speakeasy's unmarshaller sees
them: alias keys and values dereferenced, << merge keys expanded, with its precedence rules
(explicit over merged, earlier merge source over later). pureRefTarget, childByToken, and the
schema walk all route through it, and schema nodes are dereferenced at entry so an alias standing in
for a whole schema is followed.

Merge-key detection mirrors speakeasy's yml.IsMergeKey, which its marshaller applies to every
mapping it unmarshals via yml.ResolveMergeKeys. The key is examined undereferenced and its
resolved tag is checked: an alias standing in for the key is not a scalar, and a quoted '<<'
resolves to !!str. Speakeasy treats both as ordinary keys, so expanding them would invent pairs it
never sees and refuse a document that parses cleanly.

Worth recording, because it is what a future dependency bump has to re-verify against: yaml.v3's own
isMerge is not the right model even though it reads the same syntax. It is laxer about the tag
and honors only the last << in a mapping, where speakeasy merges every one. Neither difference is
reachable from a parsed document — yaml.v3 resolves plain, non-specific, and explicitly tagged <<
scalars alike to !!merge — but the scan has to agree with speakeasy, not with yaml.v3.

contentSchema joins subSchemaObjectKeys. I cross-checked the key sets field-by-field against
every *JSONSchema[Referenceable]-typed field of oas3.Schema at speakeasy v1.24.0; that was the
only one missing. The key-set doc comment now records the mapping — including the two entries
(additionalItems, definitions) that are real JSON Schema keywords the library does not type — so
a dependency bump has an explicit thing to re-verify.

The ref-collection walk is iterative, with one visited set per role. Following alias edges means
the same subtree is now reachable by several paths, so the walk needs memoization or a chained-alias
document makes it exponential — trading a crash for a hang.

Each of the four walk roles gets its own visited set rather than sharing one: an anchored $ref node
can legally be reused once where the node itself is a schema and once where its values are a
name→schema map, and a shared set let the first role mark it seen so the second skipped it — dropping
the very node the cycle chain needed. That is the sixth row above, found while reviewing the first
draft of this fix. The dispatch over roles is exhaustive, with a panicking default, so a role added
without a case fails loudly rather than being walked as whichever kind of node the switch fell
through to.

The walk is a worklist rather than a recursion because memoization and a recursion depth cap are
unsound together: a node first reached near the cap has its descent truncated and is then skipped
when a shallow path reaches it again, silently dropping every ref beneath it. A worklist has no stack
to bound, so the cap is gone and the visited sets alone bound the walk — each (node, role) pair is
enqueued exactly once. Children are pushed in reverse so the collection order stays the depth-first
pre-order a recursive walk produced, keeping the reported cycle stable for a document with more than
one.

Expansion is cached, and both the expansion and the cache are explicitly bounded. Expanding a
<< chain re-materializes every pair the levels below it contributed, so an unbounded expansion lets
a legal document cost far more than the document: without a cache a 3200-line spec took 41s, and with
one but no depth bound tighter than the walk's, a 180 KB spec with a 6000-level merge chain retained
2.2 GB while a variant ordered deepest-first took 29s. Both are the same failure this scan exists to
prevent, arriving as a hang or an OOM instead of a crash.

Two bounds keep it stated rather than open-ended:

  • maxMergeDepth (64) caps how deep a merge chain the view follows. Expanding a chain of depth d
    costs O(d²) and retains as much, so keeping d small is what makes an over-deep chain cheap to
    stop expanding rather than expensive to expand. Hand-written specs merge one or two levels and no
    generator emits more.
  • maxCachedPairs caps what the cache retains — ~50 MB at worst, whatever the input. Declining to
    cache costs a recomputation and nothing else, which is what lets the budget be a hard limit rather
    than a heuristic.

The same two documents now cost 13 MB and 0.77s, both scaling linearly with source size. For
reference, speakeasy itself is super-linear on this shape — it re-flattens each merge chain per node
it unmarshals, taking 14s on a 135 KB document — so the scan is no longer close to the dominant cost
(0.3–7% of Compile on these inputs).

Truncation is per node, not per scan. A node that hits the depth bound expands no further; every
other mapping in the document still expands in full. That distinction is load-bearing: a bound that
switched the whole view off would let a spec disable its own crash protection by carrying one
over-deep merge chain ahead of a real cycle. Dropping pairs can only make a chain terminate early or
a pointer dangle — never invent an edge — so a cycle found despite a truncation is still real and is
still reported as the error, and a clean result on a truncated scan is only "no cycle found in what
could be expanded", which refCycles says out loud as an openapi/cycle-scan-failed warning.

Only expansions that are reproducible are memoized: any complete one, and any entered at the top
level, which with nothing in flight around it is a deterministic function of the node alone. A
truncation reached from inside a deeper expansion is not — how much of the chain below survived
depends on where the walk came in — and caching that would let one traversal order lose a $ref
another would find.

walkAnchors is deliberately untouched: it must not follow alias edges structurally, since that is
exactly how it detects a recursive anchor.

Behavior change

A spec whose << merge chains nest deeper than 64 levels now carries an openapi/cycle-scan-failed
warning. It still compiles — the warning is never a refusal, and speakeasy expands such chains fine —
it only states that the pre-parse guarantee is incomplete for that source. Nothing in the conformance
corpus or the golden set reaches the bound.

Test plan

  • Six new reproducer specs under testdata/openapi/, one per shape, registered in the
    cycleReproducers table so both the detector-level test and the full-Compile test cover each.
    A green run of TestCompile_CyclicSpecDoesNotCrash is itself the proof of the fix: a fatal stack
    overflow would take the test binary with it.
  • Negative controls, since the real regression risk is refusing valid documents: legal anchored
    schema reuse, a legal << merge into a concrete schema, a legal non-cyclic contentSchema, and
    an alias-valued $ref whose chain terminates. Each must lower cleanly with no cyclic-ref
    diagnostic. TestDetectCycles_NonMergeKeyShapesAreClean adds the resolver-fidelity pair — a quoted
    '<<' and an alias-valued key — which an over-eager merge expansion would refuse.
  • TestIsMergeKey_MatchesResolver tables the tag and kind rules against speakeasy's, and
    TestIsMergeKey_AgreesWithParsedTags backs that strictness by taking tags from real parses rather
    than asserting them — showing no parsed document can reach the shapes the table rejects.
  • Whitebox tests for the mapping expansion covering alias keys and values, an alias-valued mapping,
    a key aliasing a nil target, non-scalar keys, duplicate keys, merge and merge-sequence precedence,
    a non-mapping merge value, and the depth bound from both sides (a chain exactly at it expands in
    full; one past it stops and records why).
  • TestNodeView_CachesOnlyReproducibleExpansions pins what may and may not be memoized across all
    four cases, and TestNodeView_MemoizeRespectsPairBudget pins the cache ceiling including that a
    dropped entry still reads correctly.
  • TestDetectCycles_TruncationDoesNotDisableTheRestOfTheScan and TestNodeView_TruncationIsPerNode
    pin the non-contagious rule end to end and at the unit level: a cycle declared after an over-deep
    chain is still caught, and caught as the error rather than reported as the warning.
    TestCompile_MergeChainPastBoundStillCompiles pins that the warning never costs a compile.
  • Walk-level tests: TestRefScanCollect_VisitsEachNodeOncePerRole drives one anchored $ref node
    into all three schema roles and asserts it is collected once but entered in each role;
    TestRefScanCollect_DeepNestingIsNotTruncated nests a $ref past the former depth cap and requires
    it still be collected; TestRefScanCollect_UnhandledRolePanics pins the exhaustive dispatch.
  • Three timing guards, all of which would have failed at some point during this change: a
    chained-alias fan-out document (exponential without the visited sets), a merge chain at the depth
    bound (must stay clean and cached), and a 1600-level chain with every level a schema and the
    schemas ordered deepest-first — the shape that took 29s before the bound, which must now finish
    fast and report the warning rather than claim to be clean.
  • seedCorpus now also seeds every testdata/openapi/*.yaml, so the degenerate shapes are mutation
    starting points for FuzzCompile rather than fixtures the fuzzer never sees.
  • gofmt -l ., go vet ./..., golangci-lint run (0 issues), go test ./..., and
    ./scripts/check-coverage.sh (100.0% total, 100.0% every package) all pass.

Closes #26.

…re-scan

The pre-parse cycle scan reads the raw yaml.Node tree, while speakeasy's
resolver reads the decoded tree where aliases (keys and values) are already
dereferenced and `<<` merge keys are already expanded. Wherever the scan's
node model was narrower than the decoder's, a genuine $ref cycle could slip
past it and reach the resolver, which crashes the process with an
unrecoverable stack overflow.

Add mappingPairs, a helper that walks a mapping the way the decoder sees it:
alias keys and values dereferenced, `<<` merge keys expanded, with the same
key-precedence rules yaml.v3 applies. Route pureRefTarget, childByToken, and
the schema-ref walk through it, and dereference schema nodes themselves so an
alias standing in for a whole schema is followed too. Add contentSchema to
the sub-schema key set, the one JSONSchema-typed field of oas3.Schema the
existing sets missed.

Following alias edges structurally also means the same subtree can now be
reached through more than one path, so the ref-collection walk needs
visited-node sets to keep it linear instead of letting a chained-alias
document blow up exponentially. The walk is split across four functions —
walkOutsideSchema, walkSchema, walkSchemaMap, and walkSchemaList — and each
gets its own set. A single shared set per two of them is not safe: an
anchored $ref node can legally be reused once where its own value is treated
as a schema (walkSchema) and once where its values are treated as a name-to-
schema map (walkSchemaMap), and letting the first role mark the node "seen"
made the second role skip it — silently dropping the very $ref node the
cycle chain needed and letting the resolver recurse unbounded. Each walk
function keeps its own set instead, so a node is only ever skipped on a
repeat visit in the same role.

Add reproducers for all six shapes plus legal-document controls so the scan
doesn't start refusing valid aliases and merges, and seed the fuzz corpus
with every fixture under testdata/openapi.
…rom the ref walk

Expanding `<<` merge keys made the cycle pre-scan super-linear. Every read of a
mapping recomputed the full merge closure from scratch, and the schema walk read
each node twice, so a merge chain cost O(n) per expansion and O(n) expansions:
a 3200-line document took 41 seconds. The scan exists to keep a degenerate spec
from crashing the compiler, so turning that crash into a hang is no fix.

Reads of the raw node tree now go through a nodeView, which owns the
decoder-faithful view of a mapping and memoizes each expansion for the lifetime
of one scan. Only complete expansions are cached: one truncated by a merge cycle
or by the depth cap is missing pairs that another entry point would supply, and
caching it would let traversal order decide whether a $ref is found. The same
3200-line document now scans in 90ms.

Two correctness fixes alongside it:

- Merge-key detection now matches yaml.v3's own isMerge. The key is examined
  undereferenced and its tag is checked, so a quoted '<<' (tag !!str) and an
  alias standing in for the key are ordinary keys again. Both were being
  expanded as merges, which reported a cyclic $ref in documents that parse
  cleanly and would never have reached the resolver.

- The ref-collection walk is iterative. Resolving aliases means one node is
  reachable from many parents, so the walk needs memoization to stay linear —
  but memoization and a recursion depth cap are unsound together: a node first
  reached near the cap has its descent truncated and is then skipped when a
  shallow path reaches it again, silently dropping the refs beneath it. A
  worklist has no stack to bound, so the cap is gone and each (node, role) pair
  is visited exactly once.

The per-role visited sets are now one array indexed by role rather than a field
each, so adding a role cannot leave a set nil.
The mapping view added for alias and merge-key resolution reused maxCycleDepth
(10000) as its expansion bound. Expanding a `<<` chain re-materializes every
pair the levels below it contributed, so that bound let a legal document cost
far more than the document itself: a 180 KB spec with a 6000-level merge chain
retained 2.2 GB, and one whose schemas were ordered deepest-first took 29s
where the parser the scan protects handles the same input in 20ms.

Two bounds replace it. maxMergeDepth (64) caps how deep a merge chain the view
follows, which is what makes an over-deep chain cheap to stop expanding rather
than expensive to expand — real specs merge one or two levels. maxCachedPairs
caps what the expansion cache retains, so the scan holds a stated ~50 MB at
worst whatever it is given. The same two documents now cost 13 MB and 0.77s,
both scaling linearly with source size.

Truncation is per node, not per scan. An earlier form of this latched a flag
that stopped all further expansion, which would have let a spec disable its own
cycle protection by carrying one over-deep chain ahead of a real cycle. A node
that hits the bound now simply expands no further; every other mapping still
expands in full, and refCycles reports the incompleteness as an
openapi/cycle-scan-failed warning instead of passing the scan off as clean.
Dropping pairs can only make a chain terminate early or a pointer dangle, never
invent an edge, so a cycle found despite a truncation is still real and is
still reported as the error.

An expansion entered at the top level is memoized even when truncated: with
nothing in flight around it, it is a deterministic function of the node alone,
so caching it is sound and keeps a truncated chain from re-expanding once per
node that references it.

isMergeKey now applies speakeasy's yml.IsMergeKey test rather than yaml.v3's
isMerge. Speakeasy's marshaller is what reads these documents, and the two
disagree in both directions — yaml.v3 is laxer about the tag and honors only
the last `<<` in a mapping, where speakeasy merges every one. Neither
difference is reachable from a parsed document, but the comments pointed a
future dependency bump at the wrong model to re-verify against.

The ref-collection walk's dispatch gains an explicit roleSchemaList case and a
panicking default, so a role added without a case fails loudly instead of being
walked as whichever kind of node the switch fell through to.

Behavior change: a spec whose merge chains nest deeper than 64 levels now
carries an openapi/cycle-scan-failed warning. It still compiles — the warning
is never a refusal — and nothing in the conformance corpus or golden set
reaches the bound.
@OmarAlJarrah
OmarAlJarrah merged commit 4e0c0ed into main Jul 27, 2026
1 check passed
@OmarAlJarrah
OmarAlJarrah deleted the fix/openapi-cycle-scan-alias-and-contentschema branch July 27, 2026 01:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openapi: $ref cycles through alias values and contentSchema positions crash with a fatal stack overflow

1 participant