Starcoder / Falcon LLMs do not work

[Nniol]

Hi,

We are using SmartDataFrame with OpenAI and it works well, however we want to showcase other LLMs so we are trying to use the HuggingFace LLMS

However everytime we run the code we get:
Unfortunately, I was not able to answer your question, because of the following error:\n\nNo code found in the response\n

MAX_RETRIES = 3
API_KEY = getpass.getpass("Please enter your Hugging Face API Key (hit enter): ")

llm = Starcoder(api_token = API_KEY, max_retries = MAX_RETRIES)
df_customers = pd.read_csv("aCSV.csv", index_col=0)                      
df_customers.head(3)
customers = SmartDataframe(df_customers, 
                         config={"llm": llm }
                          )
customers.chat("Where does most customer stay")

If we swap out the Starcode or Falcon to

llm = AzureOpenAI(
           api_token=api_key, api_base=api_base, api_version=api_version, deployment_name=deployment_name 
            )

It all works fine

[gventuri]

Hi @Nniol, as the prompt evolves in complexity to grant better results, unfortunately some models cannot keep the pace.
I think the real problem here is the way we have implemented the StarCoder and Falcon apis. Basically, we use the free hugging face API, that only generates a few tokens (10?) at a time. This basically forces us to loop this to do more subsequent requests trying to generate more tokens, but of course it's not the expected behavior and many things can go wrong in the process.

I wonder if it even makes sense to keep supporting this workaround. @mspronesti what's your thoughts about it?

[mspronesti]

@gventuri @Nniol I'm subjectively skeptical of the capabilities of Starcoder and Falcon. I only experienced decent/good results using OpenAI and Meta's models (Llama 2 and CodeLlama).

As the number of new issues and discussions reporting poor performances when using Starcoder and Falcon has not dropped, I'd personally deprecate them for good.

Besides, one can still use these models with pandasai via langchain, at their own "risk", whereas an "official" support implies (in my opinion) some guarantees on the performances :)

[gventuri]

@mspronesti totally, let's discontinue the other models and remove in the coming releases. The focus at the moment should be in making a library that works great with the state-of-art models, the advancements in the LLMs will make the rest!

[Nniol]

I saw no reference to Llama 2 in the documentation. Do you have an example please? ***@***.***

…

On Tue, 5 Sept 2023, 13:02 Massimiliano Pronesti, ***@***.***> wrote: @gventuri <https://github.com/gventuri> @Nniol <https://github.com/Nniol> I'm subjectively skeptical of the capabilities of Starcoder and Falcon. I only use OpenAI and Meta's models (Llama 2 and CodeLlama), which work pretty well for pandasai. As the number of new issues and discussions reporting poor performances when using Starcoder and Falcon has not dropped, I'd personally deprecate them for good. One can still use these models via langchain, at their own "risk", whereas an "official" support implies (in my opinion) some garantees on the performances :) — Reply to this email directly, view it on GitHub <#525 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOBY6GMIWP5KQGQ7SOYCJDXY4BENANCNFSM6AAAAAA4KVNFXM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mspronesti]

As no public API is currently available for Llama-2 or CodeLlama, these models are supported via huggingface's text-generation library. To try them out you need to host them yourselves, e.g. in local (assuming you have enough computing power) or in a cloud instance.

Steps to use:

• install text-generation via pip
• on your host machine start a text-generation server. The easiest way would be to run it in a docker container, as explained in their docs.
  Here's the script I usually use to run the server:

  NUM_GPUS=8
  volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
  HF_TOKEN=hf_... # your token here
  
  docker run --gpus all --shm-size 2g -p 8080:80 --env HUGGING_FACE_HUB_TOKEN=$HF_TOKEN -v $volume:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id codellama/CodeLlama-34b-Python-hf \
  --sharded true \
  --num-shard $NUM_GPUS \
  --max-input-length 4000 \
  --max-total-tokens 4096

• use pandasai.llm.HuggingfaceTextGen as llm, specifying the inference_server_url of your deployment

If you wanna try this out on AWS, notice that to use Llama 70B you need at least 8 GPUs, so make sure to have access to a 48xlarge or a p4d instance. Smaller models are less demanding, but also less powerful. For CodeLlama 34B, 4 GPUs might be enough. Hope this helps :)

[LinyeLi60]

Hi mspronesti, how many vram dose the CodeLlama 34B need? I have two a6000 with 48G vram.

[mspronesti]

@Nniol Falcon and Starcoder have been deprecated and will be removed in a future release.

[jakub-kaminiarz]

Hi !
I believe it is still possible to use it through Text Generation Inference. I was able to spin it up locally with falcon-7b, however encountered issues regarding lack of code being found in the responses.
Wanted to ask about opinion regarding Falcon/Startcoder models being used through TGI, and also are you able to suggest models that are tested and relatively responsive using TGI ?
Thank You

[lvwerra]

Hi there, I worked on StarCoder and would be happy to understand what the limitations are you are facing. Note that StarCoder is a base model trained on code and as such cannot chat. If you need a chatty model for your API maybe StarChat would solve your issues. Also note that these models run on the inference API (backed by TGI) and you can generate more than 10 tokens. You can find the chat demo code here using the inference API: https://huggingface.co/spaces/HuggingFaceH4/starchat-playground/blob/main/app.py

Also CodeLlama/Llama 2 are available via the inference API.

We are working on the next generation of StarCoder and if you have specific feedback on the model format/limitations let us know.

[mspronesti]

Hi @lvwerra, thanks a lot for your feedback.
In the past months, I've played around with Starcoder (base) and starchat-alpha (self-hosted, I couldn't run it on the inference API) on a small dataset of ~80 analytics quieres using a database containing trading data, via pandasai.
For most of the queries (>70%), they both failed at generating code at all.
I haven't tested starchat-beta.

Also, if you check the issues and discussions sections you will find a non-negligible number of threads reporting Starcoder not producing code.

As for Llama2 and CodeLlama they perform much better in my opinion, but I can't rum them on the inference API with a non-premium account, except for the 7b versions. I might be missing something though. Can you elaborate a little more on "you can generate more than 10 tokens"?

[gventuri]

@lvwerra great point, thanks for pointing that out. There are 2 things here:

• the limited 10-tokens api inference causes 95% of the errors
• the remaining 5% of the errors is caused by the base version of StarCoder itself, and IMO there are 2 solutions here
  • write a custom prompt for StarCoder to make it work better and make it less likely to hallucinate
  • use better suited models (as you correctly mentioned)

Regardless of the approach we decide, we definitely need to refactor the HF models to work TGI (@mspronesti has already implemented it recently, I guess we can wrap is for HuggingFace with the right url and settings and replace the old HuggingFaceLLM class).

And then we could list the models that are "supported", and raise a warning if other models are used mentioning that they have not be tested might not work correctly.

[lvwerra]

10-tokens api inference

just to be sure: the issue is that the API by default only returns 10 tokens? I can't reproduce that on my end. Even with the default I get more than 10 and you can set it up to 250:

curl https://api-inference.huggingface.co/models/gpt2 \
        -X POST \
        -d '{"inputs":"The answer to the universe is", "parameters": {"max_new_tokens": 250}}' \
        -H "Authorization: Bearer TOKEN" 

Result:

[{"generated_text":"The answer to the universe is not an answer to the question of why the universe exists in the first place.\n\nOn the contrary, it's an answer to the question, \"Why is something so important so important that it's worth keeping?\" The answer to the question comes straight from the creator of the universe.\n\nSome might ask this, \"What is a star?\" or \"What is a telescope?\" But a star cannot exist without a telescope and its stars are not the only objects that are there. The reason that these other objects of the universe are there is because they show up and interact with every other object that we have made so far—and thus, it wouldn't be possible to create planets without a telescope. It's also because there are some stars in every other group of things—like these spiral arms—\"that are more common than our own, because they provide a unique, stable and stable environment.\" As Newton once said, \"There are only four fundamental elements in your universe—the first is your body, the second your soul, the third your soul, the fourth your consciousness,\" (LQ 2:14). The last is the first being, which holds the world together and then the fifth being—the \"good.\" The two \""}]%

Could you share an example? Thank you!