Skip to content

DEVO-1167 Create airflow dag for gencon solr collections #1711

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

nomadicoder
Copy link
Contributor

No description provided.

@nomadicoder nomadicoder requested a review from sensei100 May 5, 2025 18:35
- Specifies ruby version for cvs indexing
- Commented out previous tries for reference for continued development
- Stores data in /tmp for indexing
- Less generic script name
@nomadicoder nomadicoder force-pushed the DEVO-1167-Create_Airflow_DAG_for_gencon_solr_collections branch from ee6ce06 to 9092780 Compare May 5, 2025 20:35
- Move multiple tasks into the ingest script
- Force csv encoding to UTF-8
- Remove development data
- Run DAG manually
- Store raw csv data in temporary directory
@nomadicoder nomadicoder marked this pull request as ready for review May 7, 2025 19:01
@nomadicoder nomadicoder force-pushed the DEVO-1167-Create_Airflow_DAG_for_gencon_solr_collections branch 3 times, most recently from 4f6faa2 to 99e8880 Compare May 15, 2025 21:08
@nomadicoder nomadicoder marked this pull request as draft May 16, 2025 13:59
- Separate gencon_index to it's own project
- Replace literal paths and URL's to environment variables
@nomadicoder nomadicoder force-pushed the DEVO-1167-Create_Airflow_DAG_for_gencon_solr_collections branch from 99e8880 to c0af430 Compare May 16, 2025 15:31
@nomadicoder nomadicoder marked this pull request as ready for review May 16, 2025 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant