Fix correctness bugs in the gender-inference path (0.9.0)

[soodoku]

Audit of naampy/in_rolls_fn.py, nnets.py, and utils.py turned up several genuine correctness bugs. Each fix has a regression test that fails against master — 10 of the 17 new tests in tests/test_regressions.py do.

Bugs fixed

Stale cache returned wrong numbers. __state and __year were declared and read in the cache-invalidation check but never assigned — only __dataset was written. So:

in_rolls_fn_gender(df, 'name', state='kerala')   # loads + filters to Kerala
in_rolls_fn_gender(df, 'name')                   # every check passes -> cache reused
                                                 # -> merges against the KERALA-ONLY table

Against real data, priya has n_female=23652 in Kerala and 41097 nationally. The second call returned Kerala's number. Replaced with a single (dataset, state, year) key, and the four filter branches collapse to filter-then-group-by-name, which also guarantees a unique merge key.

Missing names were scored as the string "nan". str(None).lower() → "nan" → encode_name → [14, 1, 14], a real model input. Rows with no name came back with a confident-looking gender. Names are now a nullable string column; rows with no usable name stay NaN throughout.

Non-Latin names were labelled "male". No a-z characters → empty encoding → the 0.5 neutral default → probs > 0.5 is False → "male" at 0.5 confidence. Every Devanagari, Gujarati, and Kannada name under an English dataset. Now None/NaN.

Re-running on naampy's own output crashed, two ways. Pre-existing columns became _x/_y merge suffixes and the prop_female lookup raised KeyError; and .at[] was handed an Index despite being a scalar-only accessor — verified on pandas 2.2.3 that it survived only via an internal KeyError fallback and raises InvalidIndexError once the column exists. Stale columns are dropped before the merge; .loc does the assignment.

The caller's DataFrame was mutated with a __first_name column that del rdf[...] only removed from the merged copy.

Truncated downloads poisoned the cache permanently. download_file streamed straight to the target, so an interrupted transfer left a short file that load_naampy_data then reported as Using cached on every later run. Now writes to a temp file and renames on success, with raise_for_status() and a Content-Length check.

Behavior changes

• pred_gender / pred_prob are now always present (except v2_native), holding None/NaN where unused. They previously appeared only when some name missed the lookup, so callers had to probe with row.get(...).
• --state is validated after parsing, against the dataset named by --dataset. It was an argparse choices= built from list_states(), so --help downloaded and parsed the full 60MB v2_1k dataset, and states were checked against v2_1k regardless of --dataset.
• CLI exits 1 rather than -1 (which the shell reports as 255).

Cleanup

• Dropped the dead find_ngrams helper (zero call sites, left over from the pre-LSTM n-gram model).
• Removed the tensorflow pin from streamlit/requirements.txt and the TF mocks from the Sphinx config. Nothing has imported TensorFlow since the PyTorch migration in 0.8.0 — streamlit_app.py imports only base64, pandas, streamlit, naampy, and uv.lock has no TF entries. Dependabot's uv ecosystem entry globs Python manifests repo-wide, which is how it kept opening PRs against that dead pin (Update tensorflow requirement from ==2.18.* to ==2.21.* #22).
• Corrected the documented state counts — v2/v2_1k carry 31 states and union territories, not 30 — and the maharastra state key, which the docs, user guide, and example notebook all spelled maharashtra, a value the data never contained.

Tests

tests/test_regressions.py is offline: it patches load_naampy_data to a small fabricated fixture with known counts, so it asserts on exact numbers rather than on whatever the live Dataverse files happen to hold. The pre-existing suite was network-bound with nearly every assertion wrapped in if pd.notna(...) guards and several bare except: pass blocks, so none of these bugs would have failed a test. Those tests are now behind NAAMPY_NETWORK_TESTS=1 and the swallowed assertions are real.

Verification

• make lint (ruff, mypy, pydoclint) — clean
• uv run pyright — 0 errors
• ruff format --check — clean
• pytest — 30 passed, 27 skipped (offline)
• NAAMPY_NETWORK_TESTS=1 pytest — 57 passed against the real datasets
• sphinx-build — succeeds with zero warnings (was 2)
• uv build — wheel and sdist build at 0.9.0

Version bumped to 0.9.0: under 0.x, a minor bump is the signal that existing callers see different output. No tag is pushed — publishing to PyPI is a separate, deliberate step.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e04646e38

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve the empty-string all-states sentinel

When the default all option is selected in streamlit/streamlit_app.py, that caller deliberately passes state=""; changing this predicate from truthiness to is not None treats the sentinel as an actual state and filters the lookup table to zero rows. Consequently, the default Streamlit flow loses all electoral-roll counts and falls back to ML for every readable name. Treat the empty string like None, or update the Streamlit caller in the same change.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Regenerate uv.lock after the version bump

The tracked uv.lock still records the editable project as naampy v0.8.0 even though this manifest now declares 0.9.0. Confirmed with uv tree --frozen—whose help describes the option as displaying requirements without locking—which reports naampy v0.8.0; frozen consumers therefore see stale resolution metadata, while non-frozen workflows must rewrite the committed lock. Regenerate and commit uv.lock with this release bump.

Useful? React with 👍 / 👎.