SDK toolchain upgrade + raw-file download, configurable scan limit, aligned randomize#1
Conversation
Run `block-tools structure refresh` to move the block onto the current canonical layout and bump the toolchain (package-builder 3.12.0 -> 3.14.1, block-tools 2.7.7 -> 2.12.2, tengo-builder -> 4.0.15, model/ui -> 1.79.x). The main motivation is package-builder 3.14, which builds the linux/amd64 software Docker image on Apple Silicon hosts (3.12 hard-gated docker builds to x64 and silently skipped them), so the block can be built for a remote k8s backend from an arm Mac. Migration fixes required by the refresh: - add a `typescript` catalog entry (the refreshed block/package.json references it via `catalog:`) - replace the empty placeholder test with a real smoke test (the upgraded linter rejects empty / .todo test files)
… randomize - Raw file download moved to a pre-run (prerun.tpl.tengo) that wraps the selected sample's original blobs with file.exportFile. The "Download raw files" button is now available as soon as a dataset + sample are chosen, with no main Run, and streams on demand without staging to a workdir. (The dataset column's data is not reachable from the model, so this must be produced workflow-side; passing the raw blob yielded "Key not found ctl/file/blobInfo" — exportFile provides the downloadable resource.) - Scan limit is now a parameter (default 2,000,000 reads), shown for the modes that scan; if it exceeds the file's read count the whole file is scanned. - Randomized range keeps paired R1/R2 mate-aligned for both gzipped and uncompressed input: both use seeded reservoir sampling, so equal-length mates pick identical ordinals. Removed the byte-offset seek path (could not preserve alignment) and its dead helpers; added a test asserting two mate files select the same ordinals. Fixes: - Correct the read-header matching tooltip (exact match, not prefix). - Drop the incorrect "R1/R2 not mates" warning for randomized gzip.
- Use the SDK app-level `progress` loader (isRunning) so the extraction shows the standard "running analysis, no action required" overlay like other blocks, instead of a custom in-page spinner. Driven by the main Run only, so raw-file pre-run downloads don't trigger it. - Block title now shows the sample's human-readable label (as picked in the dropdown) instead of the raw sample id, falling back to the id when the label can't be resolved.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request upgrades the SDK toolchain and introduces several enhancements to the FASTQ reader, including raw file downloads via a pre-run workflow without requiring a full run, a configurable scan limit, and mate-aligned randomized range sampling using seeded reservoir sampling for both compressed and uncompressed inputs. Feedback from the review highlights opportunities for defensive programming, specifically suggesting checks to prevent potential runtime crashes in the pre-run template if dataset data is undefined or keys are empty, as well as checks in the model to avoid TypeErrors if the axes specifications are empty.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| inputsMap := dataset.data.inputs() | ||
| filesByReadIndex := {} | ||
| for sKey in maps.getKeys(inputsMap) { | ||
| key := json.decode(sKey) | ||
| if string(key[0]) != args.sampleId { | ||
| continue | ||
| } | ||
| readIndex := readIndexPos >= 0 ? string(key[readIndexPos]) : "R1" | ||
| if is_undefined(filesByReadIndex[readIndex]) { | ||
| filesByReadIndex[readIndex] = inputsMap[sKey] | ||
| } | ||
| } |
There was a problem hiding this comment.
Defensive programming check:
- Since the pre-run executes automatically as soon as a dataset is selected,
dataset.datamight beundefinedif the dataset is still loading or empty. Calling.inputs()onundefinedwill cause a Tengo runtime crash. - If
keyis empty, accessingkey[0]orkey[readIndexPos]will cause out-of-bounds errors. Adding length checks prevents these crashes.
if is_undefined(dataset.data) {
return {
outputs: {},
exports: {}
}
}
inputsMap := dataset.data.inputs()
filesByReadIndex := {}
for sKey in maps.getKeys(inputsMap) {
key := json.decode(sKey)
if len(key) == 0 || string(key[0]) != args.sampleId {
continue
}
readIndex := (readIndexPos >= 0 && readIndexPos < len(key)) ? string(key[readIndexPos]) : "R1"
if is_undefined(filesByReadIndex[readIndex]) {
filesByReadIndex[readIndex] = inputsMap[sKey]
}
}
| const labels = ctx.findLabels(spec.axesSpec[0]) ?? {}; | ||
| const indices = spec ? readIndicesFromSpec(spec) : ["R1"]; | ||
| const ext = spec?.domain?.["pl7.app/fileExtension"] ?? "fastq"; | ||
| const labels = spec ? (ctx.findLabels(spec.axesSpec[0]) ?? {}) : {}; |
There was a problem hiding this comment.
Defensive programming check: if spec.axesSpec is empty, spec.axesSpec[0] will be undefined, which could cause ctx.findLabels to throw a TypeError. Adding a check for spec.axesSpec[0] ensures robustness.
| const labels = spec ? (ctx.findLabels(spec.axesSpec[0]) ?? {}) : {}; | |
| const labels = (spec && spec.axesSpec[0]) ? (ctx.findLabels(spec.axesSpec[0]) ?? {}) : {}; |
| const spec = ctx.data.inputRef | ||
| ? ctx.resultPool.getPColumnSpecByRef(ctx.data.inputRef) | ||
| : undefined; | ||
| const labels = spec ? (ctx.findLabels(spec.axesSpec[0]) ?? {}) : {}; |
There was a problem hiding this comment.
Defensive programming check: if spec.axesSpec is empty, spec.axesSpec[0] will be undefined, which could cause ctx.findLabels to throw a TypeError. Adding a check for spec.axesSpec[0] ensures robustness.
| const labels = spec ? (ctx.findLabels(spec.axesSpec[0]) ?? {}) : {}; | |
| const labels = (spec && spec.axesSpec[0]) ? (ctx.findLabels(spec.axesSpec[0]) ?? {}) : {}; |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| <div v-if="rawFileExports.length > 0" class="viewer-controls raw-bar"> | ||
| <PlBtnExportArchive | ||
| :file-exports="rawFileExports" | ||
| :disabled="rawFileExports.length === 0" | ||
| suggested-file-name="raw-files" | ||
| > | ||
| Download raw files | ||
| </PlBtnExportArchive> | ||
| </div> |
There was a problem hiding this comment.
The :disabled="rawFileExports.length === 0" binding on PlBtnExportArchive is dead code: the wrapping <div v-if="rawFileExports.length > 0"> ensures the component only renders when the array is non-empty, so the disabled condition can never be true.
Prompt To Fix With AI
This is a comment left during a code review.
Path: ui/src/pages/ReadViewer.vue
Line: 182-190
Comment:
**Redundant disabled guard inside v-if**
The `:disabled="rawFileExports.length === 0"` binding on `PlBtnExportArchive` is dead code: the wrapping `<div v-if="rawFileExports.length > 0">` ensures the component only renders when the array is non-empty, so the disabled condition can never be `true`.
How can I resolve this? If you propose a fix, please make it concise.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
R1 and R2 are separate extraction runs and finish at different times. The table keyed its columns off the reads resolved so far, so it briefly showed a single R1 column and then flipped to two when R2 arrived. Drive the column layout from the dataset's declared read indices instead, and mark a column still being processed with a "loading" tag.
1 participant
Summary
Moves the block onto the current SDK toolchain (so the
linux/amd64software image builds on Apple Silicon) and adds several read-inspection improvements plus fixes.Changes
Toolchain
Features
progress), replacing the custom spinner.Fixes
Testing
software/tests), including a new test asserting R1/R2 select identical ordinals under randomize.build:dev-remote(amd64 image pushed to public ECR) and exercised on app.research.platforma.bio: raw download without Run, extraction, aligned randomize.Release
Greptile Summary
This PR upgrades the SDK toolchain (enabling
linux/amd64Docker builds on Apple Silicon), adds raw-file download without a Run via a new pre-run, makes the read-scan limit configurable, and fixes R1/R2 mate alignment under randomized sampling by replacing byte-offset seeking with unified seeded reservoir sampling for all input types.mode_range_random) replaces the splitmode_range_random_gzip/mode_range_random_seekpaths; the same seed + equal record count guarantees identical ordinal selection on both R1 and R2 sides, fixing the mate-alignment bug. A new test (test_randomize_pairs_aligned) verifies this property.prerun.tpl.tengo) wraps each sample's original blobs asfile.exportFileresources sorawFileExportsis populated before any main Run; the model reads them viactx.prerun.resolve(...)and the UI exposes the download button immediately upon sample selection.scanCapis threaded fromBlockData→BlockArgs→ workflow params → Python, replacing the previously hardcoded2_000_000, and aPlNumberFieldcontrol is shown in the settings panel for all modes that actually scan into the file.Confidence Score: 4/5
Safe to merge with one notice worth addressing before the next release: the 'randomized over the scanned portion of the file (gzip has no random access)' warning in ReadViewer.vue is factually wrong for uncompressed files and will confuse users who hit the scan cap on plain FASTQ input.
The core algorithmic change — unified reservoir sampling replacing byte-offset seeking — is correct and well-tested by the new mate-alignment test. The pre-run wiring, model changes, and CI updates are clean. The one concrete defect is that the UI warning left over from the gzip-only era now fires for all file types, misleading users of uncompressed files. The forced importHandle cast in ReadViewer is a type-safety concern but unlikely to cause observable misbehaviour given the SDK's usage pattern.
ui/src/pages/ReadViewer.vue — stale warning text and importHandle cast; the approximate notice on line 88–92 needs updating to reflect that sampling is bounded by the scan cap for all file types, not just gzip.
Important Files Changed
approximateflag is now set for all file types when scan_cap is exceeded, but the UI warning still says "gzip has no random access".Prompt To Fix All With AI
Reviews (1): Last reviewed commit: "feat: standard running overlay and sampl..." | Re-trigger Greptile