Support JSON mode.

[MiyazonoKaori]

Any plan to integrate modules such as lm-format-enforcer. Support JSON mode.

[simon-mo]

Yes! This is our top priorities.

[hadsed]

Is there currently any work happening here? I could look into it if not.

[fullstackwebdev]

context-free grammar

it would be cool if the llama cpp GBNF grammars could be standardize so we can do stuff like the image , between different LLM frameworks.

[simon-mo]

In our roadmap, anything https://github.com/outlines-dev/outlines supports can be part of vLLM, as long as any other frameworks that support LogitsProcessors API.

[Kaotic3]

This looks good, just read through outlines - seems super useful.

When do we get it :D

[findalexli]

Have someone done benchmark on latency introduced by outlines?

[solesensei]

Is this already supported as described here #1191?

[ChuckHend]

I would also love to see this feature make it into vllm!

[wxgeorge]

Is this already supported as described here #1191?

I suspect the same.

Specifically #3211 implements handling for "request_format": { "type": "json_object" } in the completion request body, which is what I'm familiar with when we say "JSON mode".

[ChuckHend]

"request_format": { "type": "json_object" } seems to work for me. Also requires a prompt specifically asking for json response.

[DarkLight1337]

Is this already supported as described here #1191?

I suspect the same.

Specifically #3211 implements handling for "request_format": { "type": "json_object" } in the completion request body, which is what I'm familiar with when we say "JSON mode".

Yep, that should do it.