fix(nanoevents): from_parquet accepts Path/ParquetFile/file-like inputs again - #1590
fix(nanoevents): from_parquet accepts Path/ParquetFile/file-like inputs again#1590NJManganelli wants to merge 4 commits into
Conversation
…ts again The local variable fs_file was only assigned in the isinstance(file, str) branch of NanoEventsFactory.from_parquet, but it is unconditionally passed to the UprootLikeShim via openfile= at the end of the method. Every documented non-str input type (pathlib.Path, pyarrow.NativeFile / io.IOBase file-like objects, and pyarrow.parquet.ParquetFile) therefore raised UnboundLocalError. Initialize fs_file = None before the branches so only the fsspec-opened str path carries a handle to clean up; all documented input types now work and yield identical events. Adds a parametrized regression test over all four input kinds in both eager and virtual modes. Partially addresses scikit-hep#1578 (from_parquet crashes for most input types). Assisted-by: Claude Fable 5
lgray
left a comment
There was a problem hiding this comment.
🤖 AI text below 🤖
Summary
This is a clean, correct, and well-tested one-line bug fix. from_parquet documents str, pathlib.Path, pyarrow.NativeFile/io.IOBase file-likes, and pyarrow.parquet.ParquetFile as inputs, but only the str branch assigned fs_file, which is unconditionally forwarded to the shim as openfile= (factory.py:600). Every non-str input therefore crashed with UnboundLocalError. Initializing fs_file = None before the dispatch (factory.py:559) fixes it. I verified the fix is genuinely correct downstream, not just papering over the crash: the shim's read() pulls from self.file (the ParquetFile), and openfile is dereferenced only in a None-guarded __del__ (mapping/parquet.py:35-36), so openfile=None reads correctly and never closes a caller-owned handle.
Overall assessment: approve. The change is minimal, root-caused in the single shared function all callers route through, and the regression test is non-vacuous and discriminating.
Verification performed in an isolated venv:
- New test: 8 passed; full
test_nanoevents.py: 26 passed, 1 skipped (skip unrelated — nodistributed). - Discriminating proof: reverting only
factory.pyto master makes exactly 6 of 8 fail withUnboundLocalError, the 2 survivors being bothstrparams — matches the PR description exactly. - Independent equality probe (eager):
str,Path,ParquetFile, aniofile object, and apyarrow.OSFileNativeFile all produce events byte-identical to the str path (fields, fullMuon.pt, per-eventJetcounts, length=40).
Test coverage
Strong. The test is proven discriminating (fails 6/8 on master) and asserts content equality to the str reference (ak.to_list(events.Muon.pt) == ref_pt), not just absence of exceptions. It covers all documented types × eager/virtual, constructs the file object the way a user would (open(path, "rb")), and correctly handles the virtual case via ak.materialize inside the with block. Dask mode is correctly excluded (it raises NotImplementedError with a schema regardless of input type).
Nits / Optional
- Docstring omits
ParquetFile. Thefileparam docstring (factory.py:481) lists "str or pathlib.Path or pyarrow.NativeFile or io.IOBase" but notpyarrow.parquet.ParquetFile, even though it's an accepted dispatch branch (factory.py:568) and now tested. This is pre-existing, not introduced here, but since the PR is about aligning documented-vs-accepted inputs, adding it to the docstring would close the last asymmetry. - Test's file-like coverage could note
NativeFile. The matrix uses a Pythonioobject forfileobj;pyarrow.NativeFile(e.g.pyarrow.OSFile) travels the sameftypesbranch, so it's covered transitively — I confirmed it works independently. No change needed; just noting the branch is exercised.
Pydantic
Not applicable — the diff touches no pydantic code (grep -ci pydantic on the diff = 0).
Nice, tightly-scoped fix with a genuinely discriminating regression test.
Add pyarrow.parquet.ParquetFile to the docstring for from_parquet
Reorder
|
Good to go? |
|
I'm tempted to allow path-like objects in general and extend that to from_root too? What do you think? Can be done with os.fspath which supports all path-like objects (include |
|
I’m fine with extensions, what’s the venn diagram with upaths, which
ServiceX folks were interested in to support token auth forwarding
appropriately?
Best,
Nick
…On Sun, Jul 12, 2026 at 20:51 Iason Krommydas ***@***.***> wrote:
*ikrommyd* left a comment (scikit-hep/coffea#1590)
<#1590 (comment)>
I'm tempted to allow path-like objects in general and extend that to
from_root too? What do you think? Can be done with os.fspath
<https://docs.python.org/3/library/os.html#os.fspath> which supports all
path-like objects (include __fspath__)
—
Reply to this email directly, view it on GitHub
<#1590?email_source=notifications&email_token=AJDSMOVS4LYCAG35QPB2XDL5EPM4PA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJVGIZTKMRRHA22M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4952352185>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AJDSMOSVB3MIEABITVQN4AT5EPM4PAVCNFSNUABFKJSXA33TNF2G64TZHMYTKOJWG4ZTCMZZHNEXG43VMU5TIOBRGUZDMNRUGU2KC5QC>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Drop the issue-tracker reference and pre-fix bug narration from the test docstring; state what the test verifies about the current behavior. Assisted-by: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01XeYa8sEdeLGa1VX2frvoNz
|
Ah, so upath allows to embed the |
|
I threw claude at this too, but have not opened the results yet, once I do
I’ll push as a draft (but I have to have the pydantic PR merged, it’s a
base for 5 extensions).
I should have a lot to cover next coffea meeting
…On Tue, Jul 14, 2026 at 16:23 Nicholas Smith ***@***.***> wrote:
*nsmith-* left a comment (scikit-hep/coffea#1590)
<#1590 (comment)>
Ah, so upath allows to embed the storage_options in a single object? That
seems very useful.
—
Reply to this email directly, view it on GitHub
<#1590?email_source=notifications&email_token=AJDSMOSBK4ZPBBIHZACPPGT5EY65VA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJXGAZDMMRUG422M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4970262475>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AJDSMORIQUI4FAPA4CZ6EML5EY65VAVCNFSNUABFKJSXA33TNF2G64TZHMYTKOJWG4ZTCMZZHNEXG43VMU5TIOBRGUZDMNRUGU2KC5QC>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
4b9e5a9 to
a26d16f
Compare
Part of #1578 — bug 14: from_parquet crashes for most input types.
NanoEventsFactory.from_parquetdocumentsstr,pathlib.Path,pyarrow.NativeFile/io.IOBasefile-like objects, andpyarrow.parquet.ParquetFileas valid inputs, but only thestrbranch assigned the localfs_file, which is unconditionally forwarded to the parquet shim'sopenfile=argument. Every non-str input type therefore crashed withUnboundLocalError, leavingstras the only working path.This initializes
fs_file = Noneahead of the input-type dispatch, so the fsspec-materialized handle is only set (and later closed) for the str path while other input types passopenfile=None. A parametrized regression test covers all four documented input types in both eager and virtual modes and asserts they yield events identical to the str path (8 passed with the fix; 6 of 8 fail without it). Fulltests/test_nanoevents.pypasses; pre-commit clean.🤖 Generated with Claude Code