Skip to content

Switch to using new materialized tables for Athena queries and ignore Positron config files - #152

Merged
jeancochrane merged 4 commits into
mainfrom
jeancochrane/140-reduce-resource-use-of-athena-queries-in-report-generation-script
Dec 9, 2025
Merged

jeancochrane merged 4 commits into
mainfrom
jeancochrane/140-reduce-resource-use-of-athena-queries-in-report-generation-script

Conversation

@jeancochrane

@jeancochrane jeancochrane commented Dec 3, 2025 •

Copy link
Copy Markdown
Member

This PR updates the HomeVal report generation script to match the changes we made in ccao-data/data-architecture#936, which tweaked the materialization for the HomeVal data tables in our Athena data lake, and in the process renamed the tables to remove the vw_ prefix that we use to mark Athena views.

Luckily, we factored out the table names to a centralized constants module in #141, so changing the generate_homeval.py script is very simple. In addition updating the constant variables, we only need to tweak one comment and the analytics dashboard script, which is a separate R project and so doesn't use the constants file. The only tricky part of the change is that we have to alter one small part of the generate_homeval.py code where we previously assumed that all columns starting with shap_* represented numeric SHAP values, an assumption that breaks down now that we include shap_run_id as a column in the pinval.assessment_card table.

Since I was messing around with the Positron/VSCode Python debugger while debugging the shap_run_id issue, I also took a minute to fold in a quick tweak to the .gitignore file for the project to ignore all local VSCode config files. This was relevant because I created a launch.json config file in order to debug the generate_homeval.py script execution.

See here for a generate-homeval workflow run that deploys a sample PIN (01011000040000) to the staging site: https://github.com/ccao-data/homeval/actions/runs/20070650417

I also reran the analytics dashboard locally and it worked as expected.

Evidence that these are the only blocks of code across all of our repos that reference the HomeVal Athena models: https://github.com/search?q=org%3Accao-data%20pinval.vw_&type=code

@jeancochrane jeancochrane linked an issue Dec 3, 2025 that may be closed by this pull request
@jeancochrane jeancochrane changed the title Switch to using new materialized tables for Athena queries Switch to using new materialized tables for Athena queries and ignore Positron config files Dec 9, 2025
@jeancochrane
jeancochrane marked this pull request as ready for review December 9, 2025 18:41

@wrridgeway wrridgeway left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks!

@jeancochrane
jeancochrane merged commit 5d186d2 into main Dec 9, 2025
1 check passed
@jeancochrane
jeancochrane deleted the jeancochrane/140-reduce-resource-use-of-athena-queries-in-report-generation-script branch December 9, 2025 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce resource use of Athena queries in report generation script

3 participants