Skip to content

perf: score each column once on the whole file instead of averaging chunk scores - #270

Open
ThibaudDauce wants to merge 3 commits into
mainfrom
perf/eliminate-formats-during-chunking
Open

perf: score each column once on the whole file instead of averaging chunk scores#270
ThibaudDauce wants to merge 3 commits into
mainfrom
perf/eliminate-formats-during-chunking

Conversation

@ThibaudDauce

@ThibaudDauce ThibaudDauce commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

detection went from 46.5s to 107.6s on joconde.csv between 0.12.0 and main

#249 removed every way a format could be dropped before the end of the file (the early stop, the zeroing on a failed batch, zero_if_too_low), so adresse (proportion 0.55), commune and code_postal are tested against every value of every column now. On a profile of joconde, adresse._is alone is 57s over 4.2M calls.

We can't just put it back: while the file is still being read, a format below its threshold can climb back up, that's what #249 fixed. So we read first and score once at the end, from the col_values we already accumulate for the profile. The total is known by then, so giving up is exact: as soon as a format has failed on more than (1 - proportion) * total values, it's out.

joconde.csv detection
main 107.6s
this branch 68.1s

Not the 46.5s of 0.12.0, but 0.12.0 stopped reading at 910 000 rows out of 992 809 and dropped formats on the first chunk alone, which is what gave the truncated total_lines and the missed commune/code_postal on ValeursFoncieres.

Some scores change since they were an average of per-batch scores, unweighted by batch length (a 20 rows batch counted as much as a 100 000 one). Same divergences as the ones listed in #248, worth a look on your side.

test_col_val, _build_remaining_tests_per_col, _apply_proportion_thresholds and zero_if_too_low are gone, they only existed to score as we go. #249 had no tests so I added three on the new behaviour.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant