Skip to content
This repository has been archived by the owner on Dec 18, 2024. It is now read-only.

Tech report pipeline #238

Merged
merged 53 commits into from
Mar 9, 2024
Merged

Tech report pipeline #238

merged 53 commits into from
Mar 9, 2024

Conversation

giancarloaf
Copy link
Collaborator

@giancarloaf giancarloaf commented Jan 28, 2024 •
edited
Loading

Changes

  1. add new pipeline - tech_report_pipeline.py
  2. add new pipeline - tech_report_deletion.py
  3. update requirements to include firestore
  4. add tests for tech_report_pipeline.py
  5. update constants to include technology pipeline queries

Validation

  1. local validation for all pipelines
  2. partial validation for pipelines using Dataflow

giancarloaf and others added 30 commits November 12, 2023 04:17
@giancarloaf
New tech_report_pipeline
e1eb9a3
@giancarloaf
clean up testing code
449231b
@giancarloaf
update to make queries more flexible
685116e
@giancarloaf
add tech_report_deletion.py
f49c940
@giancarloaf
cleanup tech_report_deletion.py
a479045
@maceto
downgrade firestore version to solve dependency conflict
88a2fff
@maceto
new pipeline where id is generated
f7bebd1
@giancarloaf
add queries and keys to constants.py
dd7de1c
@giancarloaf
updates to tech_report_pipeline
834951d 
* add id hashing logic from tech_report_pipeline_inserts.py

* update query builder

* modify date filtering logic

* change from batched to individual Firestore inserts

* Add custom TechReportPipelineOptions class
@maceto
fix query age weight
721924a
@giancarloaf
sort keys for id hashing
85a704b
@giancarloaf
parameterize query builder
f755623
@giancarloaf
Firestore write to retry on any Exception
b274fc5
@giancarloaf
fix pipeline argument parsing
61afe5d
@giancarloaf
linting
95d2443
@giancarloaf
update page_weight query in constants
89624d7
@giancarloaf
add date filtering by month
775b244
@giancarloaf
WriteToFirestoreDoFn yields hash_id, element
e8c77cc
@giancarloaf
extract pipeline arg parsing
6a47b09
@giancarloaf
optional debug logging
01fc722
@maceto
added a field in techonologies query
546fc62
@maceto
deleted test file tec_report_pipeline_inserts
a07858b
@giancarloaf
fix debug logging for firestore ids
f50ff43
@giancarloaf
Update requirements
b25f298
@giancarloaf
Update dependency installation in workflows
5f58545
@giancarloaf
Update Apache Beam version to 2.52.0
468098f
@giancarloaf
Simplify date filtering in technology queries
35087a3
@giancarloaf
Update google-cloud-firestore version
75674e4
@giancarloaf
Add validation for missing keys in technology_hash_id function
9940e84
@giancarloaf
Add unit tests for technology_hash_id function
396ec86
@giancarloaf
Remove unnecessary code for local testing
f32b412
@giancarloaf
Linting
8923e9f
@giancarloaf
Update import statement for google.cloud.firestore in tech_report_del…
e99eae7 
…etion.py
@giancarloaf
Update import statement for google.cloud.firestore in tech_report_pip…
9954c47 
…eline.py
@giancarloaf
Update google-cloud-firestore version to 2.14.0
0f56757
@giancarloaf
Add Python 3.8 setup and dependency installation to linting action
94d4965
@giancarloaf
testing: Add check for library path
4ef9c8d
@giancarloaf
Fix import statement for Google Cloud Firestore
24487a4
@giancarloaf
Update import statements in tech_report_deletion.py and tech_report_p…
7b70987 
…ipeline.py
@giancarloaf
Fix import formatting in tech_report_deletion.py and tech_report_pipe…
ebd0fef 
…line.py
@giancarloaf
Remove Python setup and dependency installation from linting action
9292a2a
@giancarloaf
Make required parameters optional
b5fa43a
@giancarloaf
Merge branch 'main' into tech_report_pipeline
8ee2eaa
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Mar 9, 2024

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)		Lines missing
  modules
  constants.py 116
  tech_report_pipeline.py 30-35, 39-52, 59-63, 67-69, 73-75, 80-81, 130-136, 141-170, 174-178
Project Total  

This report was generated by python-coverage-comment-action

@giancarloaf giancarloaf requested a review from maceto March 9, 2024 18:59
@giancarloaf giancarloaf marked this pull request as ready for review March 9, 2024 18:59
@giancarloaf giancarloaf requested a review from rviscomi March 9, 2024 19:00
maceto
maceto approved these changes Mar 9, 2024
View reviewed changes
Copy link
Collaborator

@maceto maceto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@giancarloaf giancarloaf merged commit 5f98e52 into main Mar 9, 2024
9 checks passed
@giancarloaf giancarloaf deleted the tech_report_pipeline branch March 9, 2024 19:04
@giancarloaf giancarloaf mentioned this pull request Mar 13, 2024
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@maceto maceto maceto approved these changes

@rviscomi rviscomi Awaiting requested review from rviscomi

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@giancarloaf @maceto