Isaac - #8
Open
isaacdonnell wants to merge 13 commits into
Open
Conversation
isaacdonnell
commented
Jan 9, 2026
isaacdonnell
commented
Collaborator
- WIP: Isaac branch snapshot
- Add E1 MSA context support and batching updates
Implements the ParEGO (randomized augmented-Chebyshev scalarization) selection
path from the MOCE/FolDE hybrid spec, plus the data loading, metrics, and
simulation harness needed to evaluate it against baselines on ProteinGym DMS
replay.
New modules:
folde/parego.py scalarization, simplex weight sampling, and
constant-liar batch selection over a low-rank
factored covariance (Cov = D^T D / S + s2 I).
The dense (N,N) form is O(q^2 N^2) per slate and
measured 137 s at N=4000; factored is 0.17 s at
N=6789.
folde/multiobjective_data.py aligned multi-objective dataset loading. Resolves
embeddings from explicit local paths only, never
the fuzzy foldydata matcher, which mis-selects
Profluent E1-300M (1024-dim) when asked for
ESMC-300M (960-dim) because "e1_" sorts first.
folde/multiobjective_metrics.py Pareto utilities and quantile-anchored
hypervolume. measured_hypervolume and
diagnostic_predicted_hypervolume are separate
callables: hypervolume is scale-sensitive and
Bradley-Terry scores are ordinal on an
unidentified latent scale, so the predicted form
is a diagnostic only (spec 10.3).
Scope limits, made to fail loudly rather than silently:
ObjectiveSpec rejects goal=minimize/maintain and target_interval. These are in
the spec but no isotonic calibrator or feasibility model exists, and callers
build an all-maximize direction vector, so accepting them would optimize a
minimize objective in the wrong direction. The unused pair_threshold field was
removed rather than left advertising unimplemented spec 5.4 pair filtering.
Campaign harness gives every arm the identical candidate pool. The baseline arms
route through parego_select at M=1 rather than util.constant_liar_sample, whose
MAX_POINTS_TO_CONSIDER=5000 cap would otherwise let the parego arm search ~6.7k
candidates while the baselines saw 5000. Equivalence at M=1 is covered by
test_parego.py::test_m1_equivalence_with_constant_liar_sample.
KCNJ2 results (2 objectives, 8 rounds x 24, 3 sims, regret = HV* - HV):
parego 0.00606 +/- 0.00095 9.3/11 Pareto points
fixed_weight 0.00792 +/- 0.00071 6.7/11
single_obj 0.01265 +/- 0.00427 6.3/11
random 0.01739 +/- 0.00920 5.7/11
Ordering matches the spec's prediction, but n=3 gives a minimum achievable
two-sided p of 0.10, so this is a smoke test and not evidence at alpha=0.05.
Fixed-weight leads for rounds 1-4 before ParEGO overtakes at round 5, so the arm
ranking is budget-dependent and the round count must be fixed in advance.
61 tests passing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015PJFVWxx2jm2n2TfP64PiL
Lifts the multi-objective registry from three datasets to six. Three is
below the paired Wilcoxon signed-rank floor: the minimum achievable
two-sided p is 0.25 at n=3 and 0.0625 at n=5, so neither can clear
alpha=0.05 at any effect size. Six reaches 0.0312.
- 260730_submit_new_dataset_jobs.py creates folds for KCNE1/RASK/OXDA and
enqueues ESMC-300M embedding + E1-600M naturalness jobs, plus ESMC-300M
embeddings for the higher-order SPG1_STRSG_Wu_2016 set on the existing
SPG1_Olson fold (same wild-type; naturalness is a per-substitution table
that higher-order variants score against additively). Dry run by default,
and it validates every seq_id against the wild-type before submitting.
seq_ids are canonicalized through allele_set_to_seq_id, which sorts
alleles by position. The older notebooks/jacob/*_seqids.txt files used
mutant.replace(":", "_"), which preserves the DMS csv's arbitrary order;
for multi-mutant sets like RASK the two disagree and the embedding would
silently fail to join against the activity frame.
- 260730_consolidate_new_embeddings.py copies worker output into the exact
local paths _load_embedding_df requires, verifying the 960-dim ESMC
signature and seq_id coverage first.
- Registry tests now gate strictness on EXPECTED_SHARED_VARIANT_COUNTS:
listed datasets must load, and those whose ESM jobs are still in flight
skip instead of failing. KCNE1 has landed (N=2312) and is already strict.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015PJFVWxx2jm2n2TfP64PiL
load_multiobjective_dataset intersected its index against the naturalness table. Naturalness is a per-single-substitution table (L x 20), so that intersection silently discarded every multi-mutant: RASK loaded 2,948 of its 27,814 variants (89% lost), leaving a singles-only dataset whose Pareto front had 3 points and whose HV* was 0.999998 -- effectively no signal for any acquisition comparison. Nothing scores higher-order variants additively; no such expansion exists anywhere in the codebase. The correct behavior is the one folde.data.get_proteingym_dataset already uses: keep the variant and leave naturalness NaN. This is safe because naturalness has exactly one consumer. TorchMLPFewShotModel.pretrain masks NaN rows explicitly (has_naturalness_data = ~naturalness_series.isna()), and .predict accepts a naturalness_df but never reads it -- predictions come from embeddings alone, which are per-sequence and present for every retained variant. RASK: N 2,948 -> 23,072, Pareto front 3 -> 8, naturalness coverage 12.8% (its singles). The five singles-only datasets are bit-identical before and after, since for them the intersection was always a no-op. Adds a regression test asserting multi-mutants survive with NaN naturalness while singles keep theirs, plus a guard that raises when naturalness coverage is zero -- that means the file resolved to the wrong protein rather than a legitimately sparse table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015PJFVWxx2jm2n2TfP64PiL
Adds --only to the consolidation script and stops it reading multi-GB sources in full. The SPG1_STRSG_Wu_2016 source is 3.3 GB (149k rows x ~19 KB of embedding text); pd.read_csv on the whole file materialized ~3 GB of Python strings only to check one dimension and one key set. Reading `usecols=["seq_id"]` plus a single header row keeps it at a few MB regardless of source size, and --only avoids re-copying sources that already landed. Verified: 149,361 rows, 960-dim, 100% seq_id coverage. Also records the 12-round x 10-sim KCNJ2 sweep, whose paired Wilcoxon shows parego vs fixed_weight at p=0.85 (median diff +0.00004, 5/10 wins) -- indistinguishable, so spec criterion 7 is not met on KCNJ2. The earlier 3-sim run suggested otherwise and was a false positive. parego does beat single_obj (p=0.0098) and random (p=0.0059). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015PJFVWxx2jm2n2TfP64PiL
SPG1_STRSG_Wu_2016 has 121k quadruple mutants but only four mutated positions, so it measures local epistasis in one neighborhood and cannot show whether a model generalizes to long-range multi-mutants. Mutation order is therefore the wrong screening statistic; distinct mutated positions and sequence separation between co-mutated sites are the right ones. Surveys all 217 substitution DMS files. 69 contain multi-mutants. The survey identifies GFP_AEQVI_Sarkisyan_2016 (233 positions, median 130-residue span, order up to 15) and the three Somermeyer_2022 GFP homologs as the genuinely distributed libraries, and shows that several large multi-mutant sets (HIS7_Pokusaeva order-28, CAPSD_AAV2S order-28) are segment-local despite their size. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015PJFVWxx2jm2n2TfP64PiL
Adds five distributed multi-mutant libraries to the job submission and
consolidation scripts, selected by the survey in
260730_survey_multimutant_datasets.py.
SPG1_Wu's 121k quadruples occupy four mutually contacting positions, so
it measures local epistasis only. These five test what it cannot:
GFP_AEQVI_Sarkisyan_2016 52k variants, 233/238 positions, med span 130
Q8WTC7_9CNID_Somermeyer_2022 34k, 237 positions, order up to 43
D7PM05_CLYGR_Somermeyer_2022 25k, 234 positions
Q6WV12_9MAXI_Somermeyer_2022 31k, 221 positions
PHOT_CHLRE_Chen_2023 168k, only 15 positions but ~106-residue
spans in a 118-residue LOV domain
The three Somermeyer sets are GFP homologs, so they also support
cross-homolog transfer -- long-range generalization across sequence
identity, which no single dataset measures. PHOT is the distance control:
failing it while passing Sarkisyan isolates separation from site count.
Folds 182-186; all seq_ids validated against their wild-types before
submission.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015PJFVWxx2jm2n2TfP64PiL
The signed-rank test is the acceptance criterion, but it was being run ad hoc outside the script. Moves it in, alongside a note on why the pairing is valid (every arm replays sim i from the same seed and initial set) and on the p-value floor of 2/2^n that drives the dataset-count requirement. PTEN (12 rounds x 10 sims) is weaker than KCNJ2: parego separates from nothing, not even random (p=0.32), and single_obj edges it out on the median. Combined with KCNJ2's parego-vs-fixed_weight p=0.85, criterion 7 is unmet on both datasets measured so far. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015PJFVWxx2jm2n2TfP64PiL
Five-dataset Tier-1 study (spec 10.3/11.10) at the low-N operating point: 16 mutants/round, 5 rounds, init 16, 10 sims per arm per dataset. Per-dataset mean HV regret (lower is better): dataset front rho parego fixed_wt single random KCNJ2 11 +0.307 0.01869 0.02097 0.02728 0.04656 PTEN 6 +0.373 0.03136 0.02555 0.03421 0.04929 S22A1 7 +0.776 0.00612 0.00957 0.00963 0.02572 KCNE1 10 +0.060 0.02452 0.04000 0.03355 0.04908 OXDA 5 +0.232 0.01982 0.02257 0.02034 0.03698 Wilcoxon signed-rank on per-dataset paired differences (n=5): fixed_weight 4/5 p=0.4375 95%CI=[-0.00580, +0.01548] single_obj 5/5 p=0.0625 95%CI=[+0.00052, +0.00903] random 5/5 p=0.0625 95%CI=[+0.01716, +0.02787] Findings: - Model-based acquisition beats random selection on every dataset (47/50 individual sims), with a CI well clear of zero. - ParEGO probably beats single-objective greedy, but the effect is ~5x smaller and its CI lower bound nearly touches zero. - ParEGO vs fixed-weight Chebyshev is NOT demonstrated. The CI spans zero and is 5x wider than the point estimate. Acceptance criterion 7 asserts a fixed weight vector is insufficient; measured hypervolume cannot distinguish the two after five datasets. Either the criterion needs a coverage/spacing metric rather than hypervolume to be testable at all, or it overstates the case. - The pre-registered front-size hypothesis is REFUTED. It predicted fixed_weight >= parego on OXDA (smallest front, size 5); parego won. PTEN remains a single unexplained reversal, not a trend. - No claim can reach alpha=0.05: the two-sided signed-rank floor at n=5 is 2/2^5 = 0.0625. Spec 10.10's dataset floor of 5 is arithmetically unable to satisfy acceptance criterion 10 and must rise to >= 6. The pre-registration records which datasets it actually preceded (KCNE1 and OXDA only; the other three were a pilot whose results were seen first), so the pooled test is exploratory rather than confirmatory. It also lists six known deviations from 11.10, including the missing naturalness-only arm and the unresolved variance clamp at parego.py:447. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015PJFVWxx2jm2n2TfP64PiL
Sixth dataset completes the paired set. At n=6 the two-sided signed-rank floor is 2/2^6 = 0.03125, so alpha=0.05 is arithmetically reachable for the first time. Wilcoxon on per-dataset paired differences (n=6): fixed_weight (primary) 5/6 p=0.3125 95%CI=[-0.00250, +0.00947] NOT SIG single_obj 6/6 p=0.0312 95%CI=[+0.00168, +0.00881] sig random 6/6 p=0.0312 95%CI=[+0.01695, +0.02621] sig BH at FDR=0.10 over the two exploratory arms: both pass. Acceptance criterion 10 is NOT met on the pre-registered primary metric. ParEGO vs fixed-weight Chebyshev remains undemonstrated after six datasets and 240 simulations, with a CI that spans zero. The failure is consistency, not effect size. single_obj's median difference (+0.0041) is only 1.6x larger than fixed_weight's (+0.0025) and it reaches significance; the design resolves effects of that magnitude. fixed_weight is 5/6 because PTEN reverses, and one reversal caps the signed-rank statistic at p=0.3125 at n=6 regardless of the other five margins. More datasets tighten the CI but will not fix a "usually better, occasionally worse" effect under a sign-based test. RASK also validates spec 6.6.4's factored low-rank covariance at scale: N=23,072 is 4.6x the MAX_POINTS_TO_CONSIDER=5000 cap and completed in 45 minutes. The dense path would have needed a 23,072^2 covariance rebuilt 16 times per round. Runtime is dominated by retraining 16 MLPs on <=96 measured points, which is constant across datasets, not by the O((S+q)N) acquisition. Caveat: RASK's regret is ~10x tighter than the other datasets (0.0015 vs 0.02-0.03) because 96 measurements against 23,072 candidates leaves every arm near HV*=0.999983, so its paired observation carries little range. Results are recorded in a separate RESULTS.md rather than appended to the pre-registration, which is left unamended. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015PJFVWxx2jm2n2TfP64PiL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.