Skip to content

Python bindings for XetSession.new_range_upload() - #951

Open
lhoestq wants to merge 15 commits into
huggingface:mainfrom
lhoestq:mutate-file
Open

Python bindings for XetSession.new_range_upload()#951
lhoestq wants to merge 15 commits into
huggingface:mainfrom
lhoestq:mutate-file

Conversation

@lhoestq

@lhoestq lhoestq commented Aug 27, 2026

Copy link
Copy Markdown
Member

Required for huggingface/huggingface_hub#4752
Follows #717

This PR is mostly LLM generated (I'm sorry I'm still too slow at writing rust :x). I tried to make sure it follows the similar coding pattern as for file uploads, and I made sure it has good test coverage. I double checked it myself, wrote some tests myself, and used it a lot to build the python PR in huggingface_hub and it works great. But some aspects require a bit more reviews like progress and abort / cancel handling.

The CI needs a HF_TOKEN secret for some of the e2e tests to work with the Hub (using bucket https://huggingface.co/buckets/hf-internal-testing/test-xet-core), let me know who i should ping to take care of this

Or if you are ok with only keeping the e2e tests with local CAS and removing the e2e ones with the Hub


Note

Medium Risk
New CAS upload/composition path with auth and cancel/progress behavior; mistakes could corrupt composed files or mishandle tokens, though patterns mirror existing upload commits and tests cover main edit shapes.

Overview
Adds dirty/range upload support end-to-end so clients can patch an existing CAS file by uploading only changed byte ranges instead of re-uploading the whole object.

Rust (xet_pkg) introduces XetSession::new_range_upload() with XetRangeUploadCommit / XetRangeUploadEdit (edit, insert, delete, append, commit/abort, progress hooks). Commit sorts pending edits and calls upload_ranges.

xet_data::upload_ranges now accepts an optional shared FileUploadSession so range uploads can report progress through the same session the commit holds.

Python (hf_xet) wires session.new_range_upload(original_hash, original_size, …) to XetRangeUploadCommit / XetRangeUploadEdit, including context-manager auto-commit/abort, optional progress callbacks, and GIL-friendly blocking commit. New pytest coverage exercises edit/insert/delete/append and multi-edit (including unsorted edit order) against a local CAS.

Also ignores **/__pycache__/ and extends lockfile/dev-deps for Hub-backed Rust e2e tests (HF_TOKEN).

Reviewed by Cursor Bugbot for commit 653f33e. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread xet_pkg/src/xet_session/range_upload_edit.rs
Comment thread xet_pkg/src/xet_session/range_upload_commit.rs
Comment thread hf_xet/src/py_range_upload_commit.rs
Comment thread hf_xet/src/py_range_upload_commit.rs
@lhoestq
lhoestq marked this pull request as draft August 27, 2026 14:38
@lhoestq
lhoestq marked this pull request as ready for review August 31, 2026 16:16

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit fe15ef0. Configure here.

Comment thread hf_xet/src/py_range_upload_edit.rs
assert_eq!(report.file_info.file_size, Some(11));

Ok(())
} No newline at end of file

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orphaned range-upload test file

Low Severity

range_upload_commit_tests.rs is never declared as a module, so it is not compiled or run. The file also would not build or pass if included: upload_file is async but invoked without await, commit_blocking is called twice, and the delete assertion expects size 6 instead of 8.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit fe15ef0. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant