Skip to content

[Xet] Basic shard creation #1633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open

[Xet] Basic shard creation #1633

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
3b5e2b9
createXorbs also outputs file hash, sha256 and representation
coyotte508 Jul 16, 2025
3127a01
createXorbs handles a stream of blobs
coyotte508 Jul 16, 2025
099cc40
basic shard creation
coyotte508 Jul 16, 2025
8b35869
shard magic tag
coyotte508 Jul 16, 2025
6676431
remove shard key expiry
coyotte508 Jul 16, 2025
3fc715b
actually make API calls to xet backend to upload shards/xorbs
coyotte508 Jul 18, 2025
7a38fc7
fix prefix for shard upload
coyotte508 Jul 18, 2025
947a926
fix verificaiton data
coyotte508 Jul 21, 2025
9cd2e66
update wasm bindings
coyotte508 Jul 21, 2025
bf8ae1c
fixup! update wasm bindings
coyotte508 Jul 21, 2025
70f0a0e
no need to compute shard hash client-side
coyotte508 Jul 21, 2025
431949b
no need for shard hash
coyotte508 Jul 23, 2025
515cae6
Merge remote-tracking branch 'origin/main' into shard-creation
coyotte508 Jul 24, 2025
adbe363
add va prefix to xorb too
coyotte508 Jul 24, 2025
2f9d4a0
progress events for uploading xorbs
coyotte508 Jul 24, 2025
a13da79
commit leftover xorb after all chunks have been processed
coyotte508 Jul 24, 2025
3cbde5e
integrate xet upload in commit function
coyotte508 Jul 24, 2025
651f6c0
add local dedup for xet uploads
coyotte508 Jul 24, 2025
f3e190f
move chunk caching to its own class/file
coyotte508 Jul 24, 2025
06dcd6d
Make sure to not OOB when writing shards
coyotte508 Jul 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/hub/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
"test": "vitest run",
"test:browser": "vitest run --browser.name=chrome --browser.headless --config vitest-browser.config.mts",
"check": "tsc",
"build:xet-wasm": "./scripts/build-xet-wasm.sh -t bundler -c -b hoytak/250714-eliminate-mdb-v1"
"build:xet-wasm": "./scripts/build-xet-wasm.sh -t bundler --clean"
},
"files": [
"src",
Expand Down
1 change: 1 addition & 0 deletions packages/hub/scripts/build-xet-wasm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,7 @@ fi

# copy the generated hf_xet_thin_wasm_bg.js to the hub package and hf_xet_thin_wasm_bg.wasm to the hub package
cp "$CLONE_DIR/$PACKAGE/pkg/hf_xet_thin_wasm_bg.js" "./src/vendor/xet-chunk/chunker_wasm_bg.js"
cp "$CLONE_DIR/$PACKAGE/pkg/hf_xet_thin_wasm_bg.wasm.d.ts" "./src/vendor/xet-chunk/chunker_wasm_bg.wasm.d.ts"
echo "// Generated by build-xet-wasm.sh" > "./src/vendor/xet-chunk/chunker_wasm_bg.wasm.base64.ts"
echo "export const wasmBase64 = atob(\`" >> "./src/vendor/xet-chunk/chunker_wasm_bg.wasm.base64.ts"
base64 "$CLONE_DIR/$PACKAGE/pkg/hf_xet_thin_wasm_bg.wasm" | fold -w 100 >> "./src/vendor/xet-chunk/chunker_wasm_bg.wasm.base64.ts"
Expand Down
Loading
Loading