[SPARK-53544][PYTHON] Support complex types on observations

[ueshin]

What changes were proposed in this pull request?

Supports complex types on observations.

Why are the changes needed?

The observations didn't support complex types.

For example:

>>> observation = Observation("struct")
>>> df = spark.range(10).observe(
...     observation,
...     F.struct(F.count(F.lit(1)).alias("rows"), F.max("id").alias("maxid")).alias("struct"),
... )

• classic

>>> df.collect()
[Row(id=0), Row(id=1), Row(id=2), Row(id=3), Row(id=4), Row(id=5), Row(id=6), Row(id=7), Row(id=8), Row(id=9)]
>>> observation.get
{'struct': JavaObject id=o61}

• connect

>>> df.collect()
Traceback (most recent call last):
...
pyspark.errors.exceptions.base.PySparkTypeError: [UNSUPPORTED_LITERAL] Unsupported Literal 'struct {
...

Does this PR introduce any user-facing change?

Yes, complex types are available on observations.

>>> df.collect()
[Row(id=0), Row(id=1), Row(id=2), Row(id=3), Row(id=4), Row(id=5), Row(id=6), Row(id=7), Row(id=8), Row(id=9)]
>>>
>>> observation.get
{'struct': Row(rows=10, maxid=9)}

How was this patch tested?

Added the related tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[ueshin]

cc @heyihong