Parameter data model and validation layer

[amotl]

Thoughts

omniload mangles a bunch of sets of parameters, for example from pipeline element wrappers to their implementations, or parameter bunches like reader or writer hints. Currently, they don't adhere to some model or validation logic. Python Enums or Pydantic models+validators could be used for that going forward?

References

• Incremental data loading behaviour for local and remote files #137 (comment)
• Filesystem: Improve parameter validation/selection for write-disposition strategy (--incremental-strategy, --full-refresh) #188
• Filesystem: Properly forward omniload reader hints to Polars read_xyz kwargs #214

[amotl]

FastELT

FastELT is a wrapper around dlt that seems to use Pydantic for some details already.

ingestr

Hasn't someone from the ingestr/Python community already started a fork to work on this on another day?

[amotl]

Let's use attrs and cattrs instead of Pydantic. While I only used them for laying out a data model backbone when conceiving Tikray, and not for conversion or validation purposes yet, I was very pleased about their ergonomy.

https://threeofwands.com/why-i-use-attrs-instead-of-pydantic/