vLLM request crashes if cfg is specified
#534
Labels
Comments
viktor-ferenczi
changed the title
cfg request parameter crashes the vLLM requestcfg is specified
11 tasks
|
Root cause is that |
|
I'm closing as I reverted this PR. #541 is another attempt at adding grammar-guided generation to Outlines. What do you use grammar-guided generation for? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
Describe the issue as clearly as possible:
vLLM request crashes with
500 Internal Server Errorif thecfgrequest parameter is specified.Steps/code to reproduce the bug:
Follow the vLLM tutorial:
https://outlines-dev.github.io/outlines/reference/vllm/
Since the
outline[serve]installs old versions, install vLLM 0.2.6 and latest outline, plus pydantic 2.0 using pip. (See the actual list of package versions below.)Host this model (24GB VRAM):
deepseek-ai/deepseek-coder-6.7b-instructCommand:
Request URL:
http://127.0.0.1:8000/generateSend a request with this body:
{ "prompt": "You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n\nYou are a helpful AI assistant. You give concise answers. If you do not know something, then say so.\n### Instruction:\nWrite down the first 10 prime numbers as a comma separated list.\n\n### Response:\n", "n": 1, "best_of": 1, "presence_penalty": 0.0, "frequency_penalty": 0.0, "repetition_penalty": 1.0, "temperature": 0.0, "top_p": 1.0, "top_k": -1, "min_p": 0.0, "use_beam_search": false, "length_penalty": 1.0, "early_stopping": false, "stop": [], "stop_token_ids": [], "include_stop_str_in_output": false, "ignore_eos": false, "max_tokens": 50, "logprobs": null, "prompt_logprobs": null, "skip_special_tokens": true, "spaces_between_special_tokens": true, "cfg": "\\\n?start: DIGIT+ ( \",\" DIGIT+ )* _WS?\n%import common.DIGIT\n%import common.WS -> _WS\n" }The above request will crash with 500 Internal Server Error, see the server side exception below.
The same grammar has been tested OK with Lark without vLLM. The grammar includes optional white-space at the end, because this model seems to require it to stop.
Also, please rename the
cfgrequest parameter togrammarorlark_grammar, thanks!Error message:
Outlines/Python version information:
Context for the issue:
Due to this bug it is not possible to use a grammar with vLLM at all.
I can only use vLLM, because it has the best throughput and correctness. The GBNF grammar seems to have problems in llama.cpp and the througput is ~10x worse.
Grammar support would help avoid repeated queries due to non-conformance of LLM output, which is an essential efficiency improvement in most real-world tasks.
The text was updated successfully, but these errors were encountered: