Integrating and using Narwhals

[OutSquareCapital]

This could make dataframely more extensible but also potentially make it a generic declarative schema for anything supported by narwhals.

https://narwhals-dev.github.io/narwhals/

However this would meant a lot of refactor accross the codebase, and a whole new vision for this library (from polars to most dataframes libs).
I'm also still thinking about a way to integrate it smartly, for my own repo here for example:

https://github.com/OutSquareCapital/framelib

Happy to hear your toughts!

EDIT:
source code at:
https://github.com/OutSquareCapital/framelib/tree/master/src/framelib/_duckdb

quick example

# in the class Schema, with a _schema of dict[str, Column]

    @classmethod
    def cast(cls, df: IntoLazyFrameT) -> IntoLazyFrameT:
        return (
            nw.from_native(df)
            .lazy()
            .select([col.col.cast(col.dtype) for col in cls._schema.values()])
            .to_native()
        )

def test_nw() -> None:
    import duckdb
    import polars as pl

    class Foo(fl.duck.Schema):
        a = fl.duck.Float32()
        b = fl.duck.Float32()
        c = fl.duck.String()

    df = pl.DataFrame({"a": [1, 2], "b": [3.0, 4.0], "c": ["x", "y"]})

    df_duck: duckdb.DuckDBPyRelation = duckdb.sql("""SELECT * FROM df""")

    df_casted: pl.DataFrame = Foo.cast(df.lazy()).collect()
    duck_casted = Foo.cast(df_duck).execute()
    print("polars casted:")
    print(df_casted)
    print("duckdb casted:")
    print(duck_casted)

output:

polars casted:
shape: (2, 3)
┌─────┬─────┬─────┐
│ a   ┆ b   ┆ c   │
│ --- ┆ --- ┆ --- │
│ f32 ┆ f32 ┆ str │
╞═════╪═════╪═════╡
│ 1.0 ┆ 3.0 ┆ x   │
│ 2.0 ┆ 4.0 ┆ y   │
└─────┴─────┴─────┘
duckdb casted:
┌───────┬───────┬─────────┐
│   a   │   b   │    c    │
│ float │ float │ varchar │
├───────┼───────┼─────────┤
│   1.0 │   3.0 │ x       │
│   2.0 │   4.0 │ y       │
└───────┴───────┴─────────┘

[borchero]

Hey @OutSquareCapital, thanks for your thoughts! We have already thought about using narwhals but actively decided against it to be able to integrate with polars as seamlessly as possible. A switch to narwhals would mean, for example, that all users would need to write narwhals expressions for rules instead of using polars expressions. Another argument is that we are currently working on validation within a lazy frame. This currently requires a custom polars plugin and, thus, ties dataframely to polars. Hence, we currently have no interest in broadening the scope of this project.