Skip to content

feat: pickle in native byteorder and pass byteorder through pickling and unpickling#3893

Open
ikrommyd wants to merge 5 commits intoscikit-hep:mainfrom
ikrommyd:byteorder-in-pickling
Open

feat: pickle in native byteorder and pass byteorder through pickling and unpickling#3893
ikrommyd wants to merge 5 commits intoscikit-hep:mainfrom
ikrommyd:byteorder-in-pickling

Conversation

@ikrommyd
Copy link
Collaborator

@ikrommyd ikrommyd commented Feb 24, 2026

If one pickles and unpickles in-memory in a big-endian system, it currently requires two copies. We prevent that by always pickling in the native byteorder. Also in that case, a pickle file created in a system of opposite endianess, will be unpickled incorrectly. We encode the endianess information of the system that the pickle was created inside the pickle so that the system doing the unpickling, knows what byteorder to use when interpreting the bytes.

@ikrommyd ikrommyd marked this pull request as draft February 24, 2026 00:14
@github-actions
Copy link

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR3893

@codecov
Copy link

codecov bot commented Feb 24, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.63%. Comparing base (a6c3b25) to head (48403fa).

Additional details and impacted files
Files with missing lines Coverage Δ
src/awkward/_pickle.py 70.14% <100.00%> (-8.80%) ⬇️
src/awkward/highlevel.py 78.52% <100.00%> (ø)

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ikrommyd
Copy link
Collaborator Author

ikrommyd commented Feb 24, 2026

This has to be a new "schema 2" in order to be able to unpickle files pickled with previous versions of awkward as the unpickling function is accessed by module and function name.

@ikrommyd ikrommyd requested a review from ariostas February 26, 2026 17:56
@ikrommyd ikrommyd marked this pull request as ready for review February 26, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants