Skip to content

Commit 3c96c4f

Browse files
committed
Update dataloading.py
1 parent 23f5ce4 commit 3c96c4f

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

dataloading.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ def _load_data_shard(file: Path):
1414
tokens = torch.empty(num_tokens, dtype=torch.uint8, pin_memory=True)
1515
f.seek(256 * 4)
1616
nbytes = f.readinto(tokens.numpy())
17-
assert nbytes == 2 * num_tokens, 'number of tokens read does not match header?'
17+
assert nbytes == num_tokens, 'number of tokens read does not match header?'
1818
return tokens
1919

2020

0 commit comments

Comments
 (0)