[WIP] Minimal Tokenizer Implementation

[snico432]

Overview

In response to #262 and #263, and building off of #271, I've been working on
a minimal tokenizer for exo. Initially aimed to remove the transformers dependency,
but discovered it's a transitive dependency of mlx_lm, which will require more extensive changes.

So far I've tested on:

• llama-3.2-1b
• qwen-2.5-coder-1.5b
  (both with MLX inference engine)

Questions

1. Cut down exo package size #263 mentions removing both Jinja2 and tiktoken dependencies. Since my implementation
   is currently a wrapper around these packages, I wanted to confirm if removing these
   dependencies is still the desired direction before proceeding with changes?
2. Should we tackle the transformers dependency in a separate PR given its
   transitive nature through mlx_lm?

[AlexCheema]

This is awesome! Much needed addition.
I'm going to assign a $500 retrospective bounty for this if we can get a Minimal Tokenizer implementation working for all models without any dependency on tokenizers.

To answer your questions:

1. Jinja2 and tiktoken are fine to keep.
2. Yeah we can tackle getting rid of transformers completely in a separate PR.