ValueError: Unsupported model type t5

[JunxiChhen]

System Info

Linux gaudi2-wsf-test 5.15.0-92-generic #102-Ubuntu SMP Wed Jan 10 09:33:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

docker image: ghcr.io/huggingface/tgi-gaudi:2.0.0

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Doc claims that flan-t5 is supported. However, when I try to run "google/flan-t5-base", shows:

ValueError: Unsupported model type t5

cmd:

docker run --rm  -p 9000:8000 --runtime=habana -e HABANA_VISIBLE_DEVICES=all  -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host ghcr.io/huggingface/tgi-gaudi:2.0.0 --port 8000 --trust-remote-code --model-id google/flan-t5-base --dtype bfloat16 --max-input-tokens 1024 --max-total-tokens 2048

Expected behavior

Either add support of flan-t5 or remove this model from doc.

[regisss]

Doc claims that flan-t5 is supported

Which doc are you talking about? This one: https://huggingface.co/docs/text-generation-inference/supported_models ?

[JunxiChhen]

https://github.com/huggingface/tgi-gaudi/blob/habana-main/docs/source/supported_models.md

[regisss]

Arf yes, this is the source code of TEI's docs which is inherited in this fork. I'll add a banner in the README to specify that this doc doesn't fully apply to this fork.