Prototypes of Hail-style genomics pipelines built on DataFusion, and the benchmarks that time them. The point of the repo is comparing how different formulations of the same query plan perform, so the shape of a plan is itself a first-class subject here, not just its results.
Pipeline: A named end-to-end query over genomics data, from input tables through to its consumed result. It runs to completion; its result is never deferred work. Avoid: job, query (too narrow), driver
Combiner run: One execution of a combiner against a dataset, from resolved settings through to an Outcome. Avoid: invocation, command
Action: What a caller asks a combiner run to do with its combined rows: write them, write them and record the run, run them only until their throughput settles, collect them for display, render the plan, or execute and render the analyzed plan. The Action determines the run's Outcome. Avoid: mode (taken by Compression mode), ending, sink (the operator the rows end in, which the action chooses; a separate term)
Measured write: The action that writes a formulation's rows and records how its plan ran: a run record and the run metrics, beside the output. The rows written are the rows a plain write writes. See ADR 0016. Avoid: profiled write, benchmark run, instrumented write
Throughput probe: The action that runs a formulation's plan, writing its rows or only draining them, takes progress samples as it goes, and stops once its estimate of steady-state throughput settles or its measurement window closes. It records the run like a measured write, plus its progress samples; any rows it wrote are incomplete and are not kept. Avoid: benchmark run, early-stopped write, timed run
Shadow probe: A throughput probe that evaluates when it would stop without stopping, and runs to completion, so the estimate it would have stopped with can be checked against its estimate over the longest measurement window the dataset allows. Avoid: dry run, calibration run
Outcome: What a pipeline hands back after it runs: rows written, collected batches, or a plan rendered as text. A measured write's outcome also names what it could not record. Avoid: result (too broad), output (ambiguous with a written artifact)
Run record: The row describing one combiner run: its resolved settings and its whole-run measurements. Avoid: run row, run metadata, config row
Run metrics: The per-operator, per-partition counters and timers a plan accumulated while executing, recorded as one table. Distinct from the run record, which describes the run rather than its plan. Avoid: profile, trace, stats, analyze output
Metrics directory: The directory measured writes and throughput probes record runs under: a run record table, a run metrics table, and a probe's progress samples table, beside the checks, replay baselines and replays tables that a replay of its shadow probes writes and rewrites. Each table holds one file per run. Avoid: metrics path, output metrics, results directory
Sink: The operator a combiner run's rows end in, and the plan's only consumer: a file sink when the action writes, a partitioned file sink when it writes one file per locus interval, a collecting sink when it collects, a draining sink when it only analyzes. The action chooses the sink; the sink requires the combiner's ordering. See ADR 0014 and ADR 0015. Avoid: consumer, writer, terminal, output
Combiner: A merge of input tables into a single table in a row ordering, named by the table it produces. A combiner says what is produced, not how, and its row ordering is part of what it produces; the reference combiner and the allele combiner are the two we have. Avoid: merger, joiner
Formulation: One way of building a combiner's plan. Two formulations of the same combiner return the same rows and differ in plan shape; comparing them is the point of the repo. Avoid: variant, strategy, combiner (a formulation is not itself a combiner)
Plan builder: The part of a formulation that constructs its ordered frame over a dataset and stops. It does no runtime setup, writing, or execution. Avoid: query builder, factory
Ordered frame: What a plan builder returns: the combiner's rows as deferred work, paired with the stored ordering a sink must require of them and the output layout a write gives them. A run sinks an ordered frame, never a bare frame. Avoid: frame (unqualified), sorted frame (a sort is one way to satisfy the ordering; a merge tree is another), combined table
Output layout: How a write lays a formulation's rows out at the output path: one file, or one file per partition of its frame. The formulation chooses it; the action does not. Avoid: layout (unqualified), partitioning, write mode
Output path: The path a write is given. Under a one-file output layout it names the file, with the output format's extension; under a file-per-partition layout it names the directory the partition files go into, with none. Avoid: output (ambiguous with Outcome), destination, write path
Session: The DataFusion configuration and state a plan is built against. Avoid: context, config (either alone is narrower than what plan shape depends on)
Hostile session: A session whose settings let the optimizer split any file scan across several partitions. A plan shape that holds under one holds because the plan pins it, not because the settings spared it. Avoid: adversarial config, splitting session, test session (unqualified)
Plan shape: The structure of the physical plan a plan builder produces: which operators appear and how they nest. Distinct from the plan's results; two plan shapes can be equivalent in output and differ by an order of magnitude in time. Avoid: query plan (ambiguous between logical and physical), execution graph
Merge tree: The plan shape that yields a combiner's rows in its row ordering with no sort: one sort-preserving merge over each input group's union of scans, beneath a final merge of the groups. With one group it is a flat merge, one merge over every input table's scan. Avoid: nested merge, merge hierarchy, sort tree
CPU runtime: The Tokio runtime a pipeline's plan executes on. Separate from the IO runtime so that plan execution and object store requests do not compete for the same threads. See ADR 0006. Avoid: worker pool, executor, thread pool
IO runtime: The Tokio runtime object store requests run on. Avoid: network runtime, blocking pool (Tokio's own, separate, thing)
Sample:
One sequenced individual, identified by a string id such as HG00308.
Avoid: individual, subject
Sample set: Which samples a combiner run or an input table covers. A dataset's is nonempty and is the disjoint union of its input tables' sample sets; a caller may narrow it, whole input tables at a time, but neither empty nor extend it. Avoid: samples (unqualified), sample list, cohort
Input table: One table a combiner run merges, sorted in its dataset's row ordering, together with its sample set. A single-sample input table's sample is given by where it is stored and its rows do not carry it; a multi-sample input table's rows each carry their sample, and its sample set is declared in its sample annotation table. A combiner run's written output is a multi-sample input table to the runs above it. Avoid: per-sample table (now the single-sample case), source, part
Sample annotation table: The table stored beside a multi-sample input table, with one row per sample in its sample set: the declaration of that sample set, and the place per-sample data about those samples lives. Its presence marks the input table as complete. See ADR 0018. Avoid: sidecar, manifest, sample list, cols table
Single-file input table: A multi-sample input table stored as one file. A multi-file input table is a directory of files, read as one sorted scan. Avoid: one-file table, flat table
Split point:
A locus at which one locus interval ends and the next begins. j - 1 split points, strictly
increasing in locus order, define j locus intervals.
Avoid: boundary, breakpoint, cut point, partition key
Row-balanced split points:
The split points that cut a locus-sorted table into locus intervals holding as near equal row
counts as its loci allow: with n rows and j intervals, the k-th point is the locus of the row
at index floor(k * n / j). Whether they are found exactly from the rows or approximated from file
statistics is a method, not a different kind of split point.
Avoid: oracle split points, ideal split points, even split points
Locus interval: A contiguous stretch of loci in locus order, half-open: it includes its start and excludes its end. The locus intervals of a run partition every locus, so every row falls in exactly one; a plan that merges by locus interval merges every input table within each interval and writes each interval's rows to its own file. Avoid: range, region, vertical partition, genome partition
Input group: A subset of a combiner run's input tables that one merge node combines. The input groups of a run partition its input tables; a plan that merges by input group merges the groups again afterwards. Avoid: sample group (retired: a group holds input tables, not samples), batch, shard, horizontal partition
Dataset: One stored collection of input tables under a declared row ordering, identified by its path, the format of each file, and the input tables found there, single-sample and multi-sample alike. No sample is in two of its input tables. A dataset without a declared row ordering does not exist. Avoid: input, table (a dataset holds many input tables), corpus, dataset layout (retired)
Locus-sorted table: One file or a directory of files read as a single sorted table in locus order; it has no sample set. Unlike a sorted table it knows its locus representation, and unlike a dataset it holds one table rather than many. Avoid: standalone table, sorted frame, table (unqualified)
Generated table: A table whose rows a generator produces on demand rather than reading them from storage. It has no encoded form and no files, so it cannot exercise listing, schema inference, or file statistics, and it declares its own sort order rather than recovering one from statistics. Avoid: synthetic table, mock table, fake data
Dataset fixture: A small dataset written by a test, encoded in a real format, and held in an object store (backed by cloud storage, local disk, or memory) as a stand-in for a genomics dataset a run would read. Unlike a generated table, it is encoded and stored. Avoid: test data, sample data, fixture (unqualified)
Metadata-only table: A sorted table whose files exist only as declared metadata, a path, a size, and ordering statistics, with no stored bytes behind them, so it can be planned but never scanned. Unlike a generated table it has files, and unlike a dataset fixture nothing is encoded. Avoid: fake files, stub table, mock store, supplied-statistics table
Sorted table: A file-backed table whose files are taken to hold one sorted table, scanned as one ordered partition in the file order its ordering statistics recover under that assumption. It has no knowledge of samples, contigs, or genomics. See ADR 0011 for what the assumption trusts. Avoid: listing table, sample table, sorted scan
File pruning: Leaving out of a scan every file whose ordering statistics show that no row in it can satisfy a filter. It removes files from a sorted table's order without disturbing it. Avoid: partition pruning, file skipping, statistics pruning
Filter pushdown: Applying a filter inside the scan that reads its rows rather than in an operator above it. A pushed filter is exact when the scan applies all of it and inexact when an operator above must apply it again. Avoid: predicate pushdown, pushdown unqualified
Ordering statistics: The per-file minimum and maximum of each stored ordering field. The only evidence a sorted table has of where a file's rows fall. Avoid: file statistics (broader), min/max, pruning statistics
Locus: A position in the genome: a contig together with a position within it. The unit both combiners order and group by. Avoid: site, coordinate, variant (a variant is a locus plus alleles)
Contig ordinal: The integer standing for a contig, which names it and places it in locus order. The packed representation stores loci by it; a caller names split points by it. Avoid: contig index, chromosome number, contig id
Locus representation:
How one stored row records its locus. contig-position uses separate contig and position
fields; packed uses one Int64 locus field.
Avoid: encoding (ambiguous next to compression and Vortex encoding sets), locus format
Row ordering: A sequence of row components declaring the sort order that makes a plan mergeable rather than re-sorted, typically beginning with the locus and possibly followed by others such as alleles or the sample. It names no stored field, so one ordering is declarable against either locus representation, and a finer ordering satisfies a coarser requirement. Avoid: locus ordering (retired: not every component is a locus), sort key, ordering (unqualified), stored ordering (the expansion, a separate term)
Stored ordering:
The expansion of a row ordering into stored fields under the locus representation of a dataset or
a locus-sorted table: the locus becomes contig then position, or the packed locus field, and
each other component its own field.
Avoid: sort expressions, physical ordering, row ordering (the declaration, a separate term)
Reference data: The per-sample records covering loci where a sample matches the reference genome, stored as runs of adjacent loci rather than one record each. Input to the reference combiner. Avoid: ref blocks, non-variant data
Format: The encoding of one file, together with how to read or write it, its extension, and the compression modes it accepts. It does not describe how a dataset arranges rows across files; that is the dataset's row ordering and its output layout. Avoid: row ordering, output layout, codec, encoding
Progress sample: One reading, taken during a throughput probe, of how many rows its sink has received so far and when. Avoid: poll, tick, snapshot
Probe batch: A run of a throughput probe's consecutive progress samples spanning at least the batch duration, whose rate the stopping rule takes as one observation. Avoid: batch (unqualified: record batches, and Sample group avoids it), bucket, interval
Warmup: The stretch at the start of a throughput probe whose progress samples its estimate leaves out, because throughput has not yet settled. Avoid: ramp-up, burn-in
Measurement window: The stretch of a throughput probe its estimate covers: from the end of warmup to the stop, or to the first finished partition if that comes sooner. Avoid: steady-state interval (Locus interval owns "interval"), sample window
Steady-state throughput: A throughput probe's estimate: the rows its sink received per second over its measurement window. Avoid: rate, speed, throughput (unqualified)
Whole-run throughput: The rows a combiner run wrote per second, over the whole run from resolved settings to the completed write. Unlike steady-state throughput, it includes warmup. Avoid: end-to-end throughput, overall throughput
Normalized throughput:
Steady-state throughput times the natural log of the branching factor, over the thread count.
Merging N rows of S samples through a hierarchy of combiner runs costs about N·ln S over it in
thread-seconds, whatever the branching factor, so only it compares runs of different branching
factors or thread counts. The same formula normalizes whole-run throughput.
Avoid: efficiency, per-core throughput, merge throughput
Estimate interval: The uncertainty a throughput probe attaches to its steady-state throughput: the range its rate likely lies in, judged by how that rate varied over the measurement window. Avoid: confidence interval (unqualified), error bar, CI
Stopping rule: The judgement, from a throughput probe's progress samples and its settings alone, of when the probe stops and with what estimate. Avoid: stop condition, convergence test
Tightness check: The stopping rule's judgement, at the end of a probe batch, of whether warmup has ended and the estimate interval is narrower than the precision. Avoid: check (unqualified), test, poll
Stop reason: Why a throughput probe ended: its estimate settled (steady), a partition of its plan finished and closed its measurement window (completed), or it reached its time limit (capped). Avoid: exit status, termination cause
Replay: Judging a recorded throughput probe again under other settings, from its recorded progress samples, as the stopping rule would have judged it had it run with them. Avoid: simulation, rerun, re-evaluation
Compression mode: A compression choice, always relative to an output format: a codec and level for Parquet, or an encoding scheme set for Vortex. Avoid: compression codec (too narrow for Vortex), encoding (ambiguous without a format)
Thread count: The number of worker threads given to both of a pipeline's runtimes, defaulting to available parallelism. Independent of the session's target-partition setting. Avoid: parallelism, cores, degree of parallelism
Instance family:
The GCE machine family a published binary targets, such as c4d. It fixes the binary's
target-cpu, so a binary built for one family is not run on another.
Avoid: machine type (names a size within a family), platform
Build VM: The long-lived GCE instance that compiles published binaries. It holds nothing that cannot be rebuilt, and never a result. Avoid: dev box, builder
Runner: A throwaway GCE instance that runs combiner runs from a published binary and holds nothing once it is deleted. Avoid: worker, bench VM
Runner shape: The hardware a runner presents to a combiner run: a machine type and a number of threads per physical core. Runs on runners of one shape are comparable; repetitions of a shape are separate runners. Avoid: machine, configuration, instance type
Cell: One combination of combiner run settings that a campaign compares, apart from the runner shape. The runs of one cell differ only in the runner that ran them. Avoid: config, job, case
Campaign: A named body of combiner runs recorded together, in one metrics directory, for one comparison. Avoid: experiment, sweep run, batch
Published binary: A combiner binary built once for one commit, instance family and profile, and stored where every runner can fetch it. It is never overwritten, so its commit, family and profile name exactly one build. Avoid: artifact, build