-
Notifications
You must be signed in to change notification settings - Fork 6
Open
Description
There are multiple validation checks in the ingestion pipeline, and when a step fails, the process may skip records without leaving any trace. This makes it difficult to understand why failures occur and risks silently losing data. To improve this, failed ingestion attempts should be recorded in a log file with details about the record, the validation step, and the error reason. This will make debugging easier and allow us to reprocess only the failed data instead of missing it entirely. Make be create a global error record file for each year, so we can easily track.
Metadata
Metadata
Assignees
Labels
No labels