feat(data): support list[str] URI columns in download() expression

[Aydin-ab]

Today download() fetches one file per row. This adds support for a column where each cell is a list of file URLs — for example, one row holding all the frame paths of a video — and downloads every file in that row.

What it does

• A download() column can now hold a list of URLs per row, not just a single URL. The single-URL case behaves exactly as before.
• For a list cell, all the URLs across all rows are fetched together through the same shared connection pool that single-URL downloads already use, then grouped back per row. The output column holds one list of file contents per row, in the same order.
• Edge cases: an empty list stays empty, a missing or failed download becomes None in its slot, and a null cell stays null.

Why

Until now, anyone with several files per row had to write their own per-row thread pool to fetch them. Sending all the URLs through the one shared pool was about 26x faster than per-row pools in a small S3 benchmark (80 rows of 8 files each).

Compatibility

Single-URL downloads are unchanged — the list handling only runs for list-typed columns, so nothing else in Ray Data is affected.

Testing

• New test covering mixed list lengths, an empty list, a null cell, and a row mixing a valid and a missing URL.
• All existing download tests pass.
• Ran end-to-end on a multi-node cluster downloading from S3 across several nodes, with correct results.

[gemini-code-assist]

Code Review

This pull request introduces support for downloading columns where cells contain lists of URIs (e.g., list<string>), flattening them for concurrent downloading, and re-nesting the downloaded bytes back into a list<binary> column. The feedback focuses on correctness and performance optimizations when working with PyArrow: using column.flatten() in first_inner_uri to correctly respect slice offsets, leveraging vectorized PyArrow compute functions in flatten_uri_list and pa.ListArray.from_arrays in renest_downloaded_bytes to avoid inefficient Python list materialization, and using append_column instead of add_column for cleaner, more idiomatic PyArrow table manipulation.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 82c0828. Configure here.}

[cursor]

Empty block missing scalar columns

Medium Severity

When the first URI column is a list<string> type, a zero-row block no longer takes the early exit and list columns still get an empty list<binary> output appended. Later scalar URI columns hit len(uris) == 0 / not uris and continue without adding their bytes columns, so the table schema no longer matches blocks that have rows or a scalar-first empty block.

Additional Locations (1)

• python/ray/data/_internal/planner/_obstore_download.py#L457-L461

 

^{Reviewed by Cursor Bugbot for commit 82c0828. Configure here.}