bug: Binder Error in `reverse` function

[raisadz]

Running this:

import sqlframe
sqlframe.activate('duckdb')
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

session = SparkSession.builder.getOrCreate()

data = {"a": [[3, None, 2, 4, None]]}
rows = [{key: value[i] for key, value in data.items()} for i in range(len(data[next(iter(data.keys()))]))]
df = session.createDataFrame(rows)

df = df.withColumn("a_reversed", F.reverse("a"))
df.show()

outputs the following error:

duckdb.duckdb.BinderException: Binder Error: No function matches the given name and argument types 'reverse(BIGINT[])'. You might need to add explicit type casts.
        Candidate functions:
        reverse(VARCHAR) -> VARCHAR


LINE 1: ..., 2, 4, NULL])) AS "a1"("a")), "t18827629" AS (SELECT "a", REVERSE("a") AS "a_reversed" FROM "t70656272") SELECT "a...

Name: sqlframe
Version: 3.43.8

Name: duckdb
Version: 1.3.0

[eakmanrq]

This behavior difference is documented at least:

sqlframe/docs/duckdb.md

Lines 450 to 451 in 99da0b6

	* [reverse](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.reverse.html)
	* Only works on strings (does not work on arrays)

The core problem is that I don' know what the data type of the column is and DuckDB has separate functions for reversing a string vs. reversing an array. One thing I could potentially do is create a custom UDF that supports both or create a new function in SQLFrame that specifically captures the intent to sort an array but this obviously wouldn't be backwards compatible with PySpark.

[raisadz]

Thanks for the reply, if you could create a new function like listReverse, that would be great!