perf: index the seen-tracker with sibling chains and an adaptive hash spill

[pelletier]

Splitting #1088: this PR carries one optimization technique so its impact and review surface stay isolated.

What

The seen-tracker (duplicate-key and type-consistency validation) looked keys up by scanning every entry recorded after the parent — quadratic for many-key tables. find was 33% of citm_catalog decoding.

• Sibling chains: entries now keep their named children in an intrusive sibling chain, so a lookup costs at most the number of keys in its own table, with zero hashing for small and medium tables (including the ~40-key objects of converted-JSON documents).
• Adaptive hash spill: parents whose child count crosses 64 have their children moved to a shared seeded open-addressing index keyed by (parent, name), keeping pathological tables with thousands of keys O(1) per lookup. This also removes the quadratic decode DoS surface of the linear scan.
• O(1) array-table refresh: [[array table]] elements previously compacted the entry array to delete the previous element's descendants (more quadratic work). The array-table entry is now replaced by a fresh entry with a fresh id; the old descendants still exist but hang off an id no future lookup can reach.
• Arrays of scalars create no tracker entries at all (only inline tables declare keys).

Impact

Benchmarked on a dedicated linux/amd64 spot VM (t2d), go1.26.4, interleaved A/B vs the base of this PR, benchstat over 10 samples per side:

• sec/op geomean: -1.47%
• UnmarshalDataset/citm_catalog: -41.87%
• UnmarshalDataset/code: -9.79%
• Unmarshal/ReferenceFile/struct: -5.09%
• Unmarshal/ReferenceFile/map: -3.51%
• UnmarshalDataset/config: -2.48%
• Marshal/ReferenceFile/struct: -2.09%
• Marshal/SimpleDocument/struct: -1.95%
• RealWorldContainerdConfig: -1.57%
• RealWorldPyproject: +1.74%
• RealWorldViperRead: +2.41%
• Unmarshal/HugoFrontMatter: +3.08%
• RealWorldHugoFrontMatterBatch: +3.77%
• RealWorldGolangciStrict: +3.78%
• UnmarshalDataset/example: +4.70%
• Marshal/SimpleDocument/map: +5.45%
• UnmarshalDataset/twitter: +5.46%
• Unmarshal/SimpleDocument/struct: +6.75%
• Unmarshal/SimpleDocument/map: +8.08%

Full benchstat (sec/op, allocs/op)

sec/op

UnmarshalDataset/config  10.32m ± 1%  10.06m ± 3%  -2.48% (p=0.015 n=10)
UnmarshalDataset/canada  23.76m ± 2%  23.49m ± 2%  ~ (p=0.315 n=10)
UnmarshalDataset/citm_catalog  15.331m ± 2%  8.912m ± 3%  -41.87% (p=0.000 n=10)
UnmarshalDataset/twitter  3.621m ± 1%  3.819m ± 3%  +5.46% (p=0.002 n=10)
UnmarshalDataset/code  33.45m ± 1%  30.17m ± 1%  -9.79% (p=0.000 n=10)
UnmarshalDataset/example  77.52µ ± 1%  81.16µ ± 4%  +4.70% (p=0.000 n=10)
Unmarshal/SimpleDocument/struct  335.3n ± 1%  358.0n ± 1%  +6.75% (p=0.000 n=10)
Unmarshal/SimpleDocument/map  425.2n ± 1%  459.6n ± 1%  +8.08% (p=0.000 n=10)
Unmarshal/ReferenceFile/struct  40.01µ ± 4%  37.97µ ± 2%  -5.09% (p=0.001 n=10)
Unmarshal/ReferenceFile/map  31.74µ ± 2%  30.63µ ± 2%  -3.51% (p=0.000 n=10)
Unmarshal/HugoFrontMatter  5.665µ ± 3%  5.840µ ± 1%  +3.08% (p=0.001 n=10)
Marshal/SimpleDocument/struct  403.2n ± 2%  395.4n ± 2%  -1.95% (p=0.014 n=10)
Marshal/SimpleDocument/map  595.3n ± 0%  627.8n ± 1%  +5.45% (p=0.000 n=10)
Marshal/ReferenceFile/struct  20.78µ ± 2%  20.34µ ± 1%  -2.09% (p=0.007 n=10)
Marshal/ReferenceFile/map  39.77µ ± 3%  40.27µ ± 3%  ~ (p=0.247 n=10)
Marshal/HugoFrontMatter  8.091µ ± 3%  8.149µ ± 2%  ~ (p=0.123 n=10)
RealWorldContainerdConfig  40.33µ ± 1%  39.70µ ± 2%  -1.57% (p=0.007 n=10)
RealWorldViperRead  8.329µ ± 2%  8.530µ ± 2%  +2.41% (p=0.000 n=10)
RealWorldViperWrite  12.38µ ± 4%  12.50µ ± 2%  ~ (p=0.853 n=10)
RealWorldHugoFrontMatterBatch  94.76µ ± 2%  98.33µ ± 3%  +3.77% (p=0.000 n=10)
RealWorldGitleaksRules  64.45µ ± 3%  65.49µ ± 1%  ~ (p=0.052 n=10)
RealWorldPyproject  15.04µ ± 5%  15.30µ ± 1%  +1.74% (p=0.035 n=10)
RealWorldGolangciStrict  11.81µ ± 1%  12.25µ ± 2%  +3.78% (p=0.000 n=10)
geomean  46.81µ  46.12µ  -1.47%

allocs/op

UnmarshalDataset/config  70.19k ± 0%  70.19k ± 0%  ~ (p=1.000 n=10) ¹
UnmarshalDataset/canada  223.2k ± 0%  223.2k ± 0%  ~ (p=1.000 n=10) ¹
UnmarshalDataset/citm_catalog  49.98k ± 0%  49.98k ± 0%  ~ (p=1.000 n=10) ¹
UnmarshalDataset/twitter  15.78k ± 0%  15.78k ± 0%  ~ (p=1.000 n=10) ¹
UnmarshalDataset/code  129.5k ± 0%  129.5k ± 0%  ~ (p=1.000 n=10) ¹
UnmarshalDataset/example  399.0 ± 0%  399.0 ± 0%  ~ (p=1.000 n=10) ¹
Unmarshal/SimpleDocument/struct  2.000 ± 0%  2.000 ± 0%  ~ (p=1.000 n=10) ¹
Unmarshal/SimpleDocument/map  5.000 ± 0%  5.000 ± 0%  ~ (p=1.000 n=10) ¹
Unmarshal/ReferenceFile/struct  83.00 ± 0%  83.00 ± 0%  ~ (p=1.000 n=10) ¹
Unmarshal/ReferenceFile/map  194.0 ± 0%  194.0 ± 0%  ~ (p=1.000 n=10) ¹
Unmarshal/HugoFrontMatter  54.00 ± 0%  54.00 ± 0%  ~ (p=1.000 n=10) ¹
Marshal/SimpleDocument/struct  4.000 ± 0%  4.000 ± 0%  ~ (p=1.000 n=10) ¹
Marshal/SimpleDocument/map  4.000 ± 0%  4.000 ± 0%  ~ (p=1.000 n=10) ¹
Marshal/ReferenceFile/struct  4.000 ± 0%  4.000 ± 0%  ~ (p=1.000 n=10) ¹
Marshal/ReferenceFile/map  91.00 ± 0%  91.00 ± 0%  ~ (p=1.000 n=10) ¹
Marshal/HugoFrontMatter  20.00 ± 0%  20.00 ± 0%  ~ (p=1.000 n=10) ¹
RealWorldContainerdConfig  144.0 ± 0%  144.0 ± 0%  ~ (p=1.000 n=10) ¹
RealWorldViperRead  65.00 ± 0%  65.00 ± 0%  ~ (p=1.000 n=10) ¹
RealWorldViperWrite  33.00 ± 0%  33.00 ± 0%  ~ (p=1.000 n=10) ¹
RealWorldHugoFrontMatterBatch  820.0 ± 0%  820.0 ± 0%  ~ (p=1.000 n=10) ¹
RealWorldGitleaksRules  259.0 ± 0%  259.0 ± 0%  ~ (p=1.000 n=10) ¹
RealWorldPyproject  142.0 ± 0%  142.0 ± 0%  ~ (p=1.000 n=10) ¹
RealWorldGolangciStrict  48.00 ± 0%  48.00 ± 0%  ~ (p=1.000 n=10) ¹
geomean  211.1  211.1  +0.00%
¹ all samples are equal

Read in isolation: the entry struct grows (chain links + child count), which costs a few percent on tiny documents (Unmarshal/SimpleDocument/*, twitter's many small tables) while eliminating the quadratic behavior on large ones (citm −42%, code −10%). The follow-up decode-chain PRs (native containers, fused struct decoding) win the small-document cases back several times over — see the combined numbers in the #1088 close-out.

🤖 Generated with Claude Code