Skip to content

Add failure logs file for data ingestion #100

@zaeema-n

Description

@zaeema-n

There are multiple validation checks in the ingestion pipeline, and when a step fails, the process may skip records without leaving any trace. This makes it difficult to understand why failures occur and risks silently losing data. To improve this, failed ingestion attempts should be recorded in a log file with details about the record, the validation step, and the error reason. This will make debugging easier and allow us to reprocess only the failed data instead of missing it entirely. Make be create a global error record file for each year, so we can easily track.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions