Claude/python policy analytics rebuild omukj - #4
Open
amille67 wants to merge 5 commits into
Open
Conversation
Implements a full Python package (src/ppa/) that reproduces all 8 chapter case studies from the R-based Public Policy Analytics repo. Core library (1:1 R → Python helper mapping): - ppa.viz.themes: plot_theme, map_theme (R: plotTheme, mapTheme) - ppa.stats.quantiles: q5, qbr (R: q5, qBr) - ppa.raster.convert: rast_to_df (R: rast) - ppa.geo.nearest: mean_knn_distance (R: nn_function) - ppa.geo.buffers: multiple_ring_buffer (R: multipleRingBuffer) - ppa.ml.cv: cross_validate_poisson_by_group (R: crossValidate) - ppa.ml.thresholds: iterate_thresholds (R: iterateThresholds) - ppa.ml.fairness: iterate_fairness (R: iterateFairness) Chapter pipelines (Ch1-Ch8): - ch01_transit_indicators: Philadelphia TOD indicators + OLS - ch02_ugb_sprawl: Lancaster UGB ring buffer analysis - ch03_boston_prices_baseline: Boston home prices baseline ML - ch04_boston_prices_spatial: Spatial CV for Boston prices - ch05_chicago_policing_risk: Poisson risk model + LOAO CV - ch06_churn_bounce: Churn classification + threshold sweep - ch07_compas_fairness: COMPAS fairness grid analysis - ch08_rideshare_demand: Chicago rideshare spatiotemporal demand Testing: 82 unit tests (all passing), Ch01 integration smoke test CI: GitHub Actions workflow (lint, typecheck, unit tests, smoke) Config: YAML-based per-chapter config with env var overrides https://claude.ai/code/session_014QzqohZGaLVVo5BKrae6q9
- Add data/raw/DATA/ to .gitignore (data files are user-provided) - Add .gitkeep files to preserve data/ directory structure in git https://claude.ai/code/session_014QzqohZGaLVVo5BKrae6q9
- Remove unused imports (json, numpy, geopandas, pytest, math, etc.) - Sort import blocks to satisfy isort (I001) - Remove quoted type annotations (UP037) and add missing `from typing import Any` - Fix RUF005: replace list concatenation with unpacking syntax - Fix SIM108: collapse if/else into ternary in fairness.py - Apply black formatting to 18 files with style inconsistencies - Update ruff.toml and pyproject.toml to suppress ML-convention warnings (N803/N806 for X, X_train, C etc.) via per-file-ignores - Suppress B905 (zip strict=) project-wide as data code uses safe defaults https://claude.ai/code/session_014QzqohZGaLVVo5BKrae6q9
Ch07 (algorithmic fairness): - Replace incorrect df[group_col].values[-len(y_test):] group slicing with index-based train_test_split so group labels are correctly aligned to the shuffled test set (df.iloc[test_idx][group_col].to_numpy()) Ch02–Ch08 output paths: - Import chapter_output_dir from ppa.io.paths in every chapter's build_pipeline so the PPA_OUTPUT_ROOT env var is respected in the no-override fallback path, matching Ch01 behaviour CI: - Add validate-configs job that runs python -c "tomllib.loads(...)" on pyproject.toml and yaml.safe_load() on every config/chapters/*.yaml to catch parse errors before any test jobs run https://claude.ai/code/session_014QzqohZGaLVVo5BKrae6q9
Bug fixes: - iterate_fairness: when feature_cols=None, previously passed the entire DataFrame (including observed_col and group_col) to the model, risking label leakage and runtime errors. Now infers features by excluding observed_col and group_col; uses model.feature_names_in_ when present (sklearn compatibility). Raises ValueError if no columns remain. - iterate_fairness: add threshold_by > 0 validation to prevent silent empty grids from zero or negative step sizes. - qbr: rnd=True was silently treated as rnd=False, hiding caller mistakes. Now raises ValueError with a helpful message matching R's qBr semantics (R returns NULL for rnd=TRUE, which is effectively undefined). New helpers: - quantiles.q5_labels: returns (pd.Categorical, list[str]) in one call, combining q5 + qbr for consistent quintile map legend labeling. - themes.apply_subtitle / apply_caption: italic figure-level text helpers matching R plotTheme/mapTheme subtitle and caption style conventions. - themes: remove forced axes.titleweight="bold" to match R's plain titles. - maps: add edgecolor/linewidth params to choropleth_map for polygon legibility; add SEQUENTIAL_PALETTES, CATEGORICAL_TWO, CATEGORICAL_WONG colorblind-safe palette presets (Wong 2011). - plots.threshold_rate_curves: TP rate, FP rate, Accuracy vs threshold. - plots.fairness_frontier: FPR gap vs FNR gap scatter across threshold combinations from iterate_fairness output. Tests: 102 unit tests all pass; new tests cover every bug fix and helper. https://claude.ai/code/session_014QzqohZGaLVVo5BKrae6q9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.