fix(codegen): deterministic ord via single-thread fact interning

[hdz284]

Summary

ord(x) compiles to x's string-interning key (a lasso Spur, handed out in
insertion order). The binary loader reads each fact file as parallel byte-ranges,
so under -w N>1 workers intern their slices concurrently — every ord value,
and any min(ord(..)) representative built on it (e.g. the DOOP port's
HeapAllocation_Merge), becomes nondeterministic across worker counts and
even run-to-run. Only -w 1 was stable.

Fix: when the program uses ord, load facts on worker 0 alone, in the
same order the single-thread run uses; evaluation still runs on every worker.
So -w N interning order == -w 1 == Souffle, and ord (with every
min(ord(..))) is deterministic and byte-exact again. ord's value is
unchanged — no fixtures or reference outputs need re-baselining.

Why gate on ord (addresses the review on #208)

ord is the only construct that can observe interning order, so serialising
the load for every --str-intern program was unnecessary:

• equality on interned strings is a bijection (Spur eq == content eq);
• ordered string comparison emits resolve(l) < resolve(r) — resolves to content;
• min/max semirings are numeric-only, so a string extremum must go through ord.

Hence deterministic_load = str_intern_enabled() && program.uses_ord(). Every
--str-intern program that never calls ord keeps the fully parallel loader at
no cost. New Program::uses_ord() also walks loop/fixpoint blocks (a gap
the typechecker's own ord check has via Segment::as_rules).

Validation

• cargo test --release --workspace — 328 passed / 0 failed (incl. 5 new
  uses_ord unit tests). cargo fmt --check clean. Rebased on main-next.
• Full DOOP standalone re-verification — 19 families, FlowLog -w32 vs
  Souffle -j32 (compiled and run threaded), luindex, each run twice:
  19/19 byte-exact to Souffle AND run-to-run deterministic, every count ==
  the -w1 reference. The worst pre-fix family
  (1-object-1-type-sensitive+heap, which drifted ~146 K tuples run-to-run) is
  now MATCH.

	before fix (-w32)	after fix (-w32)
vs Souffle, 19 families	19/19 DIFF	19/19 MATCH
run A vs run B	nondeterministic	byte-identical
ord value / references	—	unchanged

Overhead: nil for ord-free programs; a bounded ~2 s worker-0 load for ord
programs (FlowLog -w32 timings/RSS match the pre-fix build, e.g.
context-insensitive 19.1 s / 4.5 GB). Full runtime + peak-RSS table available.

Updated from the previous single-thread-interning commit with the requested
ord-gate refinement, and rebased onto main-next.

[HarukiMoriarty]

Only when ord builtin function is used we need the deterministic loading. ord is default require to enable str-intern.

[hdz284]

Done — the deterministic load is now gated on actual ord use: deterministic_load = str_intern_enabled() && program.uses_ord(). --str-intern programs that never call ord keep the fully parallel loader. Program::uses_ord() also walks loop/fixpoint blocks. Rebased onto main-next; cargo test 328/0 and the full 19-family DOOP suite is byte-exact + deterministic at -w32.

[HarukiMoriarty]

Fix is correct and the determinism write-up is great. One structural ask before merge: the new ord-detection (flowlog-parser/src/program/builtins.rs + mod builtins; + its 5 tests) re-implements a walk flowlog-typechecker::check_builtin_config_requirements (primitive/mod.rs:38) already does — same rule/head/compare/factor traversal, already branches on BuiltinOperator::Ord, already has config in hand. Let's make "needs deterministic load" a side effect of that existing check, carried on the unified Config, and drop the standalone walker.

Guidance

1. flowlog-common::Config — add a derived flag (same category as the existing derived output_to_stdout):

pub struct Config {
    // ...
    /// Set by the typechecker when the program uses `ord(_)`. Forces the
    /// binary-mode loader to intern on worker 0 alone so `ord` keys match
    /// `-w 1`. Derived from the program, not user-supplied.
    pub serialize_load: bool,
}

// getter, next to str_intern_enabled():
pub fn serialize_load(&self) -> bool { self.serialize_load }

The 4 exhaustive Config { .. } literals each need serialize_load: false: cli.rs:80, pipeline.rs:99, and the two test literals (dedup.rs:96, planner_errors.rs:20).

2. check_program — take &mut Config (lib.rs:81) and fold detection into the one walk. Rewriting check_builtin_config_requirements as a pure "first ord span" finder + a decision at the top keeps the nested fns capture-free and does both jobs:

pub(crate) fn resolve_ord_load(
    program: &Program,
    config: &mut Config,
) -> Result<(), TypeCheckError> {
    fn arith(a: &Arithmetic) -> Option<Span> {
        factor(a.init()).or_else(|| a.rest().iter().find_map(|(_, f)| factor(f)))
    }
    fn factor(f: &Factor) -> Option<Span> {
        match f {
            Factor::Var(_) | Factor::Const(_) => None,
            Factor::FnCall(fc) => fc.args().iter().find_map(arith),
            Factor::Builtin(bc) if bc.op() == BuiltinOperator::Ord => Some(bc.span()),
            Factor::Builtin(bc) => bc.args().iter().find_map(arith),
            Factor::Cast(c) => factor(c.inner()),
            Factor::Group(a) => arith(a),
            Factor::Tuple(t) => t.exprs().find_map(arith),
            Factor::TupleProj { tuple, .. } => arith(tuple),
        }
    }

    // NOTE: chain `as_loop()` — the current loop only visits `as_rules()`
    // (line 69), which skips `loop`/`fixpoint` blocks.
    let ord_span = program.segments().iter().find_map(|seg| {
        let plain = seg.as_rules().iter();
        let looped = seg.as_loop().into_iter().flat_map(|b| b.rules().iter());
        plain.chain(looped).find_map(|rule| {
            let body = rule.rhs().iter().find_map(|p| match p {
                Predicate::PositiveAtom(_) | Predicate::NegativeAtom(_) => None,
                Predicate::Compare(c) => arith(c.left()).or_else(|| arith(c.right())),
            });
            body.or_else(|| {
                rule.head().head_arguments().iter().find_map(|h| match h {
                    HeadArg::Var(_) => None,
                    HeadArg::Arith(a) => arith(a),
                    HeadArg::Aggregation(agg) => arith(agg.arithmetic()),
                })
            })
        })
    });

    if let Some(span) = ord_span {
        if !config.str_intern_enabled() {
            return Err(TypeCheckError::OrdRequiresStrIntern { span });
        }
        config.serialize_load = true; // the side effect
    }
    Ok(())
}

Two things this fixes vs. the current function: (a) it no longer early-returns when str_intern is on (primitive/mod.rs:42-43) — that early-return is why the ord + --str-intern case would otherwise never be walked; (b) it now walks loop/fixpoint rules, which also closes a latent gap where ord inside a fixpoint with --str-intern off currently escapes OrdRequiresStrIntern. Update the call in check_program (lib.rs:85) to the new name.

3. io/input.rs:67 — consume the flag, drop the Program::uses_ord() call:

let deterministic_load = self.config.serialize_load();

str_intern is implied (the flag is only ever set under it), so both the && and uses_ord() disappear.

4. Delete flowlog-parser/src/program/builtins.rs and the mod builtins; line in program/mod.rs. Move the essential cases — positive, comparison, min(ord()) aggregation, and the fixpoint-nested regression — into a flowlog-typechecker test asserting the side effect:

let mut cfg = Config { str_intern: true, ..Default::default() };
check_program(&mut prog, &mut cfg).unwrap();
assert!(cfg.serialize_load);

5. Point the 5 check_program call sites at &mut: main.rs:40, pipeline.rs:57, common.rs:32, typechecker_errors.rs:16, program_planner.rs:129 (the &Config::default() ones become &mut Config::default()).

One nuance worth a code comment: the flag is set at typecheck, before const-fold/DCE, so a dead ord-bearing rule that's later eliminated leaves it conservatively true — an unnecessary serial load, never a wrong result.

[HarukiMoriarty]

Minor / non-blocking — just to scope the guarantee: worker-0 load pins interning order for strings that come from input files / inline facts, but ord of a string first created during evaluation (a rule-head constant like Tag("x"), or a cat/substr/to_string/UDF result) is still interned in parallel and stays order-dependent under -w N.

That's fine for the DOOP min(ord(v)) usage (all v are loaded symbols), and Soufflé doesn't guarantee it under -j N either — so nothing to fix. Might just be worth narrowing the input.rs comment's "byte-for-byte identical to -w 1" to ord of loaded/inline symbols.

[HarukiMoriarty]

@hdz284 quick reminder to sign off the commits — 76c096d (docs(runtime): …) is missing its Signed-off-by: line (82b1049 already has it). Mind adding the DCO sign-off before merge? e.g. git rebase --signoff HEAD~2 && git push -f.