Skip to content

Add hsanci to nssp #2162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Nov 10, 2025
Merged

Add hsanci to nssp #2162

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a1c2f61
add hsanci code logic
minhkhul Nov 7, 2025
80bb904
lint: make happy
dshemetov Nov 7, 2025
c5d5dcc
hsanci -> hsa-nci + add comment about min() for dedupe
minhkhul Nov 7, 2025
d51b420
order
minhkhul Nov 7, 2025
615572d
use drop_duplicates
minhkhul Nov 10, 2025
d4b2642
adjust test for new timeout
minhkhul Nov 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions nssp/delphi_nssp/constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
"state",
"county",
"hhs",
"hsanci",
]

SIGNALS_MAP = {
Expand Down
2 changes: 1 addition & 1 deletion nssp/delphi_nssp/pull.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,5 +177,5 @@ def pull_nssp_data(
# Format county fips to all be 5 digits with leading zeros
df_ervisits["fips"] = df_ervisits["fips"].apply(lambda x: str(x).zfill(5) if str(x) != "0" else "0")

keep_columns = ["timestamp", "geography", "county", "fips"]
keep_columns = ["timestamp", "geography", "county", "fips", "hsa_nci_id"]
return df_ervisits[SIGNALS + keep_columns]
5 changes: 5 additions & 0 deletions nssp/delphi_nssp/run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,11 @@ def run_module(params, logger=None):
df = geo_mapper.add_geocode(df, "state_code", "hhs", from_col="state_code", new_col="geo_id")
df = geo_mapper.aggregate_by_weighted_sum(df, "geo_id", "val", "timestamp", "population")
df = df.rename(columns={"weighted_val": "val"})
elif geo == "hsanci":
df = df[["hsa_nci_id", "val", "timestamp"]]
df = df[df["hsa_nci_id"] != "All"]
df = df.groupby(["hsa_nci_id", "timestamp"])['val'].min().reset_index()
Copy link
Contributor

@melange396 melange396 Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this for?

Copy link
Contributor

@dshemetov dshemetov Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The data source reports at the HSA-NCI level and duplicates the same value across the constituent counties. This picks out just one of those per key.

Copy link
Contributor

@melange396 melange396 Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok, for deduplicating. in that case, the .min() is misleading/confusing and could use a small explanatory comment

Copy link
Contributor

@dshemetov dshemetov Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough. Wrote a similar comment on the epidata-etl side.

Copy link
Contributor

@melange396 melange396 Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an alternative couldve been:

df = df.drop_duplicates()

df = df.rename(columns={"hsa_nci_id": "geo_id"})
else:
df = df[df["county"] != "All"]
df["geo_id"] = df["fips"]
Expand Down
Loading