MILAB-6319: canonicalize workflow params for deterministic CIDs (block-side)#84
MILAB-6319: canonicalize workflow params for deterministic CIDs (block-side)#84PaulNewling wants to merge 3 commits into
Conversation
Catalog was pinned at 2.8.0, npm latest is 2.8.2; bumping to avoid CI rejection.
Tengo's stdlib json.encode does not sort map keys, so passing raw Tengo maps to processColumn's extra.params (or directly to smart.createJsonResource) produces non-deterministic resource bytes and non-deterministic CIDs. That silently defeats cross-project dedup of pure body renders. Introduce a block-local helper, workflow/src/canonical-resource.lib.tengo, that wraps smart.createValueResource(constants.RTYPE_JSON, canonical.encode(value)). Use canonicalResource.create(map) at each affected site; consumers read the resulting JSON resource through inputs.X as a normal Tengo map (the framework auto-decodes JSON resources, so no explicit json.decode is needed in body templates). The helper pattern is the workaround recommended by the harness reflection at mictx-helper/harnesses/block-dev/_meta/reflections/processed/2026-04-29-create-json-resource-dedup-trap.md until an SDK-side fix lands. Other determinism fixes bundled in here because they touch the same files and the same class of trap: - Move referenceLibrary out of params into its own input field on mixcr-analyze and mixcr-export. References cannot round-trip through a JSON value resource, so the params resource bytes have to exclude them. - Switch codon-map iteration in mixcr-export and export-report to maps.getKeys(codonMap) so the emitted PTabler workflow JSON is byte-stable between runs. - Remove unused 'times' imports from mixcr-analyze and export-report. - For the buildLibrary mode in main.tpl.tengo, switch the library JSON encoding from json.encode to canonical.encode for the same reason: nested map literals in the library structure would otherwise emit non-canonical bytes. Scope: block only. Tests in wf.test.ts were already failing on main with CIDConflictError; this change eliminates the body-render-layer (RenderTemplate:1) conflicts but the remaining json/getField:1 conflicts stem from SDK-internal createJsonResource(map) calls in processColumn and xsv.importFile that the block cannot reach. Full resolution requires an SDK change tracked separately. See docs/text/work/ad-hoc/2026-05-21-amplicon-alignment-cid-audit.md for the full audit.
Patch bump on workflow + block packages so the canonicalization change gets released.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces canonical encoding for workflow parameters and sorted-key map iterations to ensure deterministic resource CIDs and enable reliable cross-project deduplication. Feedback on the changes identifies a critical bug where canonical.encode returns bytes instead of a string, which will break downstream type checks on referenceLibrary. Additionally, several other map iterations in export-report.tpl.tengo need to be refactored to use sorted keys to prevent remaining sources of non-determinism.
| checksumVersion: 1 | ||
| }] | ||
| referenceLibrary = string(json.encode(library)) | ||
| referenceLibrary = canonical.encode(library) |
There was a problem hiding this comment.
In Tengo, canonical.encode returns bytes. Since referenceLibrary is expected to be a string in downstream templates (which check is_string(referenceLibrary) to decide whether to write it as a file or add it as a resource), passing raw bytes will cause is_string to evaluate to false. This will incorrectly route the execution to mixcrCmdBuilder.addFile(libraryFileName, referenceLibrary), which expects a resource reference and will fail at runtime.
Wrap the canonical encoding in string() to ensure it is treated as a string, matching the previous behavior of string(json.encode(library)).
referenceLibrary = string(canonical.encode(library))
| pframes := import("@platforma-sdk/workflow-tengo:pframes") | ||
| xsv := import("@platforma-sdk/workflow-tengo:pframes.xsv") | ||
| slices := import("@platforma-sdk/workflow-tengo:slices") | ||
| maps := import("@platforma-sdk/workflow-tengo:maps") |
There was a problem hiding this comment.
While importing maps to sort codon map keys is a great step toward determinism, there are several other direct map iterations in this file that still introduce non-determinism and should be updated to use maps.getKeys() to ensure complete CID stability:
- Line 138:
for i, clnsFile in clnsFiles— Iterating overclnsFilesdirectly causes non-deterministic argument and file addition order forexportReportCmd, which leads to unstable CIDs for the command itself. - Line 183:
for key, clonesFile in chainData.inputs()— Appends tocountDfsin a non-deterministic order, resulting in non-deterministic row order in the concatenated DataFrame. - Line 217:
for key, clnsFile in clnsFiles— Appends tofilterCountDfsin a non-deterministic order. - Line 298:
for key, clonesFile in chainData.inputs()— Appends toperChainDfsin a non-deterministic order.
Consider refactoring these loops to iterate over sorted keys using maps.getKeys(...).
1 participant
What this does
Eliminates non-deterministic CID inputs in the published workflow so cross-project deduplication of the expensive MiXCR analyze / export / aggregate steps actually fires. Adds
workflow/src/canonical-resource.lib.tengoand routes theextra.paramsmaps passed topframes.processColumn(and thebuildLibraryreference encoding) throughcanonical.encode(key-sorted) instead ofjson.encode, whose Go-map iteration order varies per render. Also sorts codon-map iteration, movesreferenceLibraryinto its own input field, and bumpsblock-tools.Why
json.encodeof a Tengo map produces non-deterministic bytes (random key order), so the per-element body-render templates get a different CID each run →CIDConflictErrorand silent cross-project dedup misses. Canonical (sorted-key) encoding makes those CIDs stable.Scope and dependencies
RenderTemplate:1) conflicts.createJsonResourceand the trace string inworkflow-tengo— needed to fix the SDK-internaljson/getField:1conflicts this block can't reach. Both land together.json/getField:1conflict in the clns/export path survives both fixes and is under separate investigation (likely a tool- or data-blob-level determinism source).Testing
On a clean backend the body-render conflicts are gone and a first run of
wf.test.tspasses. Full block CID-cleanliness depends on the paired SDK PR plus the residual-trap investigation.