Preserve Complex Data Types for to_csv

[Jaspvr]

This Pull Request solves the issue outlined in:
#60895

Complex data types like numpy arrays can now be stored in csv format and later recovered.
A new parameter, preserve_complex, is introduces, and when it is set to true in your to_csv function call, the complex data types will be preserved and can be recovered from the csv.

The way this works is by serializing Numpy arrays into JSON format for preserve_complex=True. To get them from the csv, we can set the same parameter in read_csv, and the original Numpy array will be returned.

Please refer to tests in scripts/tests/test_csv.py to see how this is used.

Please refer to the original issue for more information on the problem definition.

[snitish]

@Jaspvr is there an existing issue that this PR addresses? If so, could you list it in the description? If not, please create an issue describing the bug or proposed enhancement so it can be reviewed by a team member.

[Jaspvr]

@Jaspvr is there an existing issue that this PR addresses? If so, could you list it in the description? If not, please create an issue describing the bug or proposed enhancement so it can be reviewed by a team member.

Hey, just updated the description. This is in relation to #60895

[rhshadrach]

Thanks for the PR! You may want to review our development documentation, namely this section:

https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests

[rhshadrach]

As commented in the issue, you can use the dtype argument in read_csv to read complex values already. I'm negative on this approach.

[simonjayhawkins]

CSV's fundamental design is meant for simple, flat data structures, and embedding complex objects within it starts to deviate from that principle. JSON arrays inside a CSV file may technically work, but it introduces challenges in readability, compatibility, and usability in spreadsheet software like Excel or Google Sheets.

With nearly 50 arguments, adding more requires careful consideration to avoid unnecessary complexity. Pandas is widely used in data processing, and every additional feature should be weighed against maintainability and ease of use.

Does it align with the expectations of CSV as a format that is universally readable? Is there significant demand for preserving complex structures within CSV, or are existing solutions (e.g., JSON, Parquet) sufficient?

It seems more reasonable to encourage users to store complex data types in formats like Parquet or JSON rather than forcing CSV to handle something it's not naturally designed for.

[a-holm]

This PR introduces a useful feature for preserving complex data types like NumPy arrays during CSV serialization/deserialization via the preserve_complex flag. The implementation integrates well with the existing I/O functions, and the new tests seem cover the basic functionality (I haven't run them).

Two minor points for consideration:

Serialization Logic (pandas/io/formats/csvs.py): The current logic checks only the first value in an object column to decide if it contains arrays/lists needing serialization. This might be fragile if the first row has NaN or a different type. Would checking the first non-NaN value or using pd.api.types.infer_dtype be more robust, balancing robustness vs. performance?
Deserialization Logic (pandas/io/parsers/readers.py): The _restore_complex_arrays function uses a heuristic (startswith("[") / endswith("]")) combined with checking all() non-null values. This seems safer than just checking brackets, but could still potentially misidentify columns if they contain a mix of valid JSON strings and other strings. Perhaps a try-except json.JSONDecodeError within the apply could be an alternative? (though likely slower if many non-JSON strings exist)