Add grammar to chat/completions endpoint / Messages API

[ggbetz]

Feature request

Add grammar as additional parameter to chat/completions.

Motivation

Constrained decoding with grammars is a killer feature for many users. Exposing grammar parameter would allows users to guide chat completions output with regex or json schemes.

The openai client has an extra_body argument, so that we could access TGI endpoints as follows

from openai import OpenAI
from pydantic import BaseModel, conint
from typing import List

class Animals(BaseModel):
    location: str
    activity: str
    animals_seen: conint(ge=1, le=5)  # Constrained integer type
    animals: List[str]

+grammar =  {
+    "type": "json",
+    "value": Animals.schema()
+}

client = OpenAI(
    base_url="http://localhost:3000/v1",
    api_key="_",
)

chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {
            "role": "system",
            "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.",
        },
        {
            "role": "user",
            "content": "What's the weather like the next 3 days in San Francisco, CA?",
        },
    ],
+   extra_body={"grammar"=grammar},
    max_tokens=500,
)

This exposes grammars in all frameworks based on openai client. E.g., grammars could also be used via bind(extrabody=...) in LangChain OpenAIChatModels hooked on TGI Messages API.

Your contribution

Discuss implementations, test, documentation

[Tyler-Odenthal]

Could be pretty cool, we would use it! - Arcee Team ^-^

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[aymeric-roucher]

This has normally been solved by PR #2046!