Repository navigation
Speed up allele comparisons and per-chromosome splitting - #130
Merged
apriha merged 1 commit intoOct 6, 2026
Merged
Conversation
Profiling find_shared_dna and find_discordant_snps on two individuals with ~660k SNPs showed most of the time was spent outside the genetic computation: - Comparing genotypes allele by allele with the pandas .str accessor. Alleles are now decoded once per individual into arrays of character codes and compared with NumPy (_get_alleles, _no_shared_allele, _same_genotype, _is_discordant). - Splitting SNPs by chromosome with a full comparison per chromosome, and re-filtering the already single-chromosome DataFrame in _compute_shared_dna for every saved segment. SNPs are now split once with groupby, and _compute_shared_dna uses its DataFrame directly. Results are unchanged; best of 3 runs (660k SNPs, two individuals): find_shared_dna 4.0s -> 1.2s, find_discordant_snps 1.1s -> 0.24s.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #130 +/- ##
==========================================
+ Coverage 97.04% 97.13% +0.09%
==========================================
Files 5 5
Lines 575 594 +19
Branches 76 75 -1
==========================================
+ Hits 558 577 +19
Misses 6 6
Partials 11 11 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
apriha
approved these changes
Oct 6, 2026
apriha
left a comment
Owner
There was a problem hiding this comment.
Awesome work, thank you! Really nice speedup. 👍
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Speed up allele comparisons and per-chromosome splitting
Results are unchanged; this is a pure performance change.
Where the time went
I profiled
find_shared_dnaandfind_discordant_snpson two individuals with ~660k SNPs across 22 chromosomes and a HapMap2-sized genetic map. Most of the time was outside the genetic computation:.straccessor (.str[0],.str[1]): ~60% offind_shared_dnaand ~88% offind_discordant_snps.df["chrom"] == chromscans: one full-table comparison per chromosome to split SNPs, plus_compute_shared_dnare-filtering its already single-chromosome DataFrame for every segment it saved.Changes
(n, 2)array of character codes (_get_alleles) and compared with NumPy (_no_shared_allele,_same_genotype,_is_discordant). A missing allele (single-allele calls, e.g. on X/MT) is code 0 and never matches anything, the same as the NaN produced by.str[1]before.groupby("chrom", sort=False), which keeps the original chromosome order, and_compute_shared_dnauses its DataFrame directly.Timings (best of 3, two individuals, ~660k SNPs)
find_shared_dnafind_discordant_snpsVerification
mainon 40 randomized datasets. The data covered unordered genotypes (AG/GA), single-allele calls, indel calls (ID/DI), a 3-character genotype, missing calls, X with a male individual, 2- and 3-individual comparisons, and pair and trio discordance. Every shared DNA segment table, discrepant SNP index and discordant SNP table was identical, including a run with low thresholds that produced ~19k segments per dataset.test_allele_comparisonsfor the helper edge cases. The existing tests pass.tests/test_resources.pyandtest_sibling_shared_dna_proportionsalso happen onmainagainst currentsnpsmain (Resources._download_fileno longer exists). They aren't related to this PR.This is independent of #129. If both are merged, the second one will need a small rebase around the match-mask code in
find_shared_dna, and I'm happy to do it.