Skip to content

fix(nanoevents): from_parquet accepts Path/ParquetFile/file-like inputs again - #1590

Open
NJManganelli wants to merge 4 commits into
scikit-hep:masterfrom
NJManganelli:fix/1578-from-parquet-inputs
Open

fix(nanoevents): from_parquet accepts Path/ParquetFile/file-like inputs again#1590
NJManganelli wants to merge 4 commits into
scikit-hep:masterfrom
NJManganelli:fix/1578-from-parquet-inputs

Conversation

@NJManganelli

Copy link
Copy Markdown
Collaborator

Part of #1578 — bug 14: from_parquet crashes for most input types.

NanoEventsFactory.from_parquet documents str, pathlib.Path, pyarrow.NativeFile/io.IOBase file-like objects, and pyarrow.parquet.ParquetFile as valid inputs, but only the str branch assigned the local fs_file, which is unconditionally forwarded to the parquet shim's openfile= argument. Every non-str input type therefore crashed with UnboundLocalError, leaving str as the only working path.

This initializes fs_file = None ahead of the input-type dispatch, so the fsspec-materialized handle is only set (and later closed) for the str path while other input types pass openfile=None. A parametrized regression test covers all four documented input types in both eager and virtual modes and asserts they yield events identical to the str path (8 passed with the fix; 6 of 8 fail without it). Full tests/test_nanoevents.py passes; pre-commit clean.

🤖 Generated with Claude Code

…ts again

The local variable fs_file was only assigned in the isinstance(file, str)
branch of NanoEventsFactory.from_parquet, but it is unconditionally passed to
the UprootLikeShim via openfile= at the end of the method. Every documented
non-str input type (pathlib.Path, pyarrow.NativeFile / io.IOBase file-like
objects, and pyarrow.parquet.ParquetFile) therefore raised
UnboundLocalError. Initialize fs_file = None before the branches so only the
fsspec-opened str path carries a handle to clean up; all documented input
types now work and yield identical events.

Adds a parametrized regression test over all four input kinds in both eager
and virtual modes.

Partially addresses scikit-hep#1578 (from_parquet crashes for most input types).

Assisted-by: Claude Fable 5

@lgray lgray left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI text below 🤖

Summary

This is a clean, correct, and well-tested one-line bug fix. from_parquet documents str, pathlib.Path, pyarrow.NativeFile/io.IOBase file-likes, and pyarrow.parquet.ParquetFile as inputs, but only the str branch assigned fs_file, which is unconditionally forwarded to the shim as openfile= (factory.py:600). Every non-str input therefore crashed with UnboundLocalError. Initializing fs_file = None before the dispatch (factory.py:559) fixes it. I verified the fix is genuinely correct downstream, not just papering over the crash: the shim's read() pulls from self.file (the ParquetFile), and openfile is dereferenced only in a None-guarded __del__ (mapping/parquet.py:35-36), so openfile=None reads correctly and never closes a caller-owned handle.

Overall assessment: approve. The change is minimal, root-caused in the single shared function all callers route through, and the regression test is non-vacuous and discriminating.

Verification performed in an isolated venv:

  • New test: 8 passed; full test_nanoevents.py: 26 passed, 1 skipped (skip unrelated — no distributed).
  • Discriminating proof: reverting only factory.py to master makes exactly 6 of 8 fail with UnboundLocalError, the 2 survivors being both str params — matches the PR description exactly.
  • Independent equality probe (eager): str, Path, ParquetFile, an io file object, and a pyarrow.OSFile NativeFile all produce events byte-identical to the str path (fields, full Muon.pt, per-event Jet counts, length=40).

Test coverage

Strong. The test is proven discriminating (fails 6/8 on master) and asserts content equality to the str reference (ak.to_list(events.Muon.pt) == ref_pt), not just absence of exceptions. It covers all documented types × eager/virtual, constructs the file object the way a user would (open(path, "rb")), and correctly handles the virtual case via ak.materialize inside the with block. Dask mode is correctly excluded (it raises NotImplementedError with a schema regardless of input type).

Nits / Optional

  • Docstring omits ParquetFile. The file param docstring (factory.py:481) lists "str or pathlib.Path or pyarrow.NativeFile or io.IOBase" but not pyarrow.parquet.ParquetFile, even though it's an accepted dispatch branch (factory.py:568) and now tested. This is pre-existing, not introduced here, but since the PR is about aligning documented-vs-accepted inputs, adding it to the docstring would close the last asymmetry.
  • Test's file-like coverage could note NativeFile. The matrix uses a Python io object for fileobj; pyarrow.NativeFile (e.g. pyarrow.OSFile) travels the same ftypes branch, so it's covered transitively — I confirmed it works independently. No change needed; just noting the branch is exercised.

Pydantic

Not applicable — the diff touches no pydantic code (grep -ci pydantic on the diff = 0).

Nice, tightly-scoped fix with a genuinely discriminating regression test.

Add pyarrow.parquet.ParquetFile to the docstring for from_parquet
@NJManganelli

Copy link
Copy Markdown
Collaborator Author

Good to go?

@ikrommyd

Copy link
Copy Markdown
Member

I'm tempted to allow path-like objects in general and extend that to from_root too? What do you think? Can be done with os.fspath which supports all path-like objects (include __fspath__)

@NJManganelli

NJManganelli commented Jul 13, 2026 via email

Copy link
Copy Markdown
Collaborator Author

Drop the issue-tracker reference and pre-fix bug narration from the test
docstring; state what the test verifies about the current behavior.

Assisted-by: Claude Opus 4.8
Claude-Session: https://claude.ai/code/session_01XeYa8sEdeLGa1VX2frvoNz
@nsmith-

nsmith- commented Jul 14, 2026

Copy link
Copy Markdown
Member

Ah, so upath allows to embed the storage_options in a single object? That seems very useful.

@NJManganelli

NJManganelli commented Jul 14, 2026 via email

Copy link
Copy Markdown
Collaborator Author

@NJManganelli
NJManganelli force-pushed the fix/1578-from-parquet-inputs branch from 4b9e5a9 to a26d16f Compare July 25, 2026 16:45
@NJManganelli

Copy link
Copy Markdown
Collaborator Author

This PR, however, is straightforward, can we merge? Let's leave upath and other extensions to their own PR(s). @lgray @ikrommyd @nsmith-

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants