Cache Can Break on Version Update

[lapp0]

Describe the issue as clearly as possible:

#544 doesn't work for me unless I delete the cache introduced in #492

This is because a cached functions signature has changed, resulting in this error:

>       self.states_to_token_maps, self.empty_token_ids = create_states_mapping(
            regex_string, tuple(sorted(tokenizer.vocabulary.items()))
        )
E       ValueError: too many values to unpack (expected 2)

We should either incorporate a version number into the cache or clear the cache upon user upgrade.

Reproduction steps

• checkout main
• construct a RegexFSM
• checkout Fix incompleteness of regex and cfg guided generation #544
• construct the same RegexFSM

[rlouf]

How do libraries usually do this?

[lapp0]

I can't find much prior art. I recommend we do something like

memory = Cache(cache_dir, eviction_policy="none", cull_limit=0)
if outlines.__version__ != memory.get("__version__"):
    memory.clear()
memory["__version__"] = outlines.__version__

We would then require anyone writing breaking changes to a cached function to increment Outlines patch number.