Skip to content

Speed up allele comparisons and per-chromosome splitting - #130

Merged
apriha merged 1 commit into
apriha:mainfrom
AlsammanAlsamman:feature/vectorize-allele-comparisons
Oct 6, 2026
Merged

apriha merged 1 commit into
apriha:mainfrom
AlsammanAlsamman:feature/vectorize-allele-comparisons

Conversation

@AlsammanAlsamman

Copy link
Copy Markdown
Contributor

Speed up allele comparisons and per-chromosome splitting

Results are unchanged; this is a pure performance change.

Where the time went

I profiled find_shared_dna and find_discordant_snps on two individuals with ~660k SNPs across 22 chromosomes and a HapMap2-sized genetic map. Most of the time was outside the genetic computation:

  1. Allele-by-allele genotype comparison with the pandas .str accessor (.str[0], .str[1]): ~60% of find_shared_dna and ~88% of find_discordant_snps.
  2. Repeated df["chrom"] == chrom scans: one full-table comparison per chromosome to split SNPs, plus _compute_shared_dna re-filtering its already single-chromosome DataFrame for every segment it saved.

Changes

  • Alleles are decoded once per individual into an (n, 2) array of character codes (_get_alleles) and compared with NumPy (_no_shared_allele, _same_genotype, _is_discordant). A missing allele (single-allele calls, e.g. on X/MT) is code 0 and never matches anything, the same as the NaN produced by .str[1] before.
  • SNPs are split by chromosome once with groupby("chrom", sort=False), which keeps the original chromosome order, and _compute_shared_dna uses its DataFrame directly.

Timings (best of 3, two individuals, ~660k SNPs)

before after
find_shared_dna 4.0 s 1.2 s
find_discordant_snps 1.1 s 0.24 s

Verification

  • I compared outputs against main on 40 randomized datasets. The data covered unordered genotypes (AG/GA), single-allele calls, indel calls (ID/DI), a 3-character genotype, missing calls, X with a male individual, 2- and 3-individual comparisons, and pair and trio discordance. Every shared DNA segment table, discrepant SNP index and discordant SNP table was identical, including a run with low thresholds that produced ~19k segments per dataset.
  • Added test_allele_comparisons for the helper edge cases. The existing tests pass.
  • The 7 failures in tests/test_resources.py and test_sibling_shared_dna_proportions also happen on main against current snps main (Resources._download_file no longer exists). They aren't related to this PR.

This is independent of #129. If both are merged, the second one will need a small rebase around the match-mask code in find_shared_dna, and I'm happy to do it.

Profiling find_shared_dna and find_discordant_snps on two individuals
with ~660k SNPs showed most of the time was spent outside the genetic
computation:

- Comparing genotypes allele by allele with the pandas .str accessor.
  Alleles are now decoded once per individual into arrays of character
  codes and compared with NumPy (_get_alleles, _no_shared_allele,
  _same_genotype, _is_discordant).
- Splitting SNPs by chromosome with a full comparison per chromosome,
  and re-filtering the already single-chromosome DataFrame in
  _compute_shared_dna for every saved segment. SNPs are now split once
  with groupby, and _compute_shared_dna uses its DataFrame directly.

Results are unchanged; best of 3 runs (660k SNPs, two individuals):
find_shared_dna 4.0s -> 1.2s, find_discordant_snps 1.1s -> 0.24s.
@codecov

codecov Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.13%. Comparing base (fa49eb4) to head (7fc3255).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #130      +/-   ##
==========================================
+ Coverage   97.04%   97.13%   +0.09%     
==========================================
  Files           5        5              
  Lines         575      594      +19     
  Branches       76       75       -1     
==========================================
+ Hits          558      577      +19     
  Misses          6        6              
  Partials       11       11              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@apriha apriha left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work, thank you! Really nice speedup. 👍

@apriha
apriha merged commit 50b4062 into apriha:main Oct 6, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants