We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
1 parent 529de22 commit 82facf9Copy full SHA for 82facf9
README.md
@@ -27,7 +27,7 @@ Example code using `tiktoken` can be found in the
27
28
`tiktoken` is between 3-6x faster than a comparable open source tokeniser:
29
30
-
+
31
32
Performance measured on 1GB of text using the GPT-2 tokeniser, using `GPT2TokenizerFast` from
33
`tokenizers==0.13.2`, `transformers==4.24.0` and `tiktoken==0.2.0`.
0 commit comments