[WIP] Minimal Tokenizer Implementation#513

Draft

snico432 wants to merge 11 commits intoexo-explore:mainfrom snico432:remove-transformers-dep

+216-33

Commits on Nov 24, 2024

Commits on Nov 27, 2024

Commits on Nov 28, 2024

Commits on Nov 30, 2024

simplified tokenizer by just using vocab instead of merges
snico432
committed

Commits on Dec 6, 2024

Commits on Dec 7, 2024

breaking tokenizers into seperate classes that inherit from abstract base class
snico432
committed