Local LLM using Ollama without OpenAI

[lindamathez]

Hi all,
I was trying to get started with paperqa using Ollama instead of OpenAI API - as I have limitations on the use of OpenAI. However, I have downloaded llama3.2 (llm and summary_llm) and nomc-embed-text (embedding) from Ollama.

My goal is to use only local models and to not use any OpenAI models at all.

My environment looks like this:

• python=3.11.10
• requirements.txt

My code looks like this:

from paperqa import Settings, ask

# Define the missing EnvironmentState -> was getting Pydantic errors without it
class EnvironmentState:
    def __init__(self, *args, **kwargs):
        pass

# Rebuild the Settings model -> was getting Pydantic errors without it
Settings.model_rebuild()

local_llm_config = {
    "model_list": [
        {
            "model_name": "ollama/llama3.2",  
            "litellm_params": {
                "model": "ollama/llama3.2",  
                "api_base": "http://localhost:11434",  # Ollama server API base URL
            },
        }
    ]
}

settings = Settings(
    llm = "ollama/llama3.2",
    llm_config = local_llm_config,
    summary_llm = "ollama/llama3.2",
    summary_llm_config = local_llm_config,
    embedding = "ollama/nomic-embed-text",
    paper_directory = "papers",
)

answer = ask(
    "What machine learning algorithms can classify heart failure?",
    settings = settings
)

print(answer)

I wonder how to decouple paperqa from OpenAI API. I added Ollama information for the keywords llm, summary_llm and embedding - but it still seems to be trying to access OpenAI. My error is the following:

 AuthenticationError: litellm.AuthenticationError: AuthenticationError: OpenAIException - Error code: 401 - {'error': {'message': 
 'Incorrect API key provided: foo. You can find your API key at https://platform.openai.com/account/api-keys.', 'type':           
 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}                                                              
 Received Model Group=gpt-4o-2024-08-06                                                                                           
 Available Model Group Fallbacks=None  

It seems that the code is still trying to access OpenAI despite the fact that I've provided the path to the local deployment of my Ollama models.

Many thanks for hints - Cheers :)

[dosubot[bot]]

Hey @lindamathez 🌟, I'm here to help you with your technical issues and questions. Let's tackle this bug together and get everything up and running smoothly!

To configure paperqa to use only local models from Ollama and completely decouple it from the OpenAI API, you need to ensure that all default settings pointing to OpenAI models are overridden with your local model configurations. Here's a solution based on similar issues:

1. Ensure Local Model Configuration: Your configuration for llm, summary_llm, and embedding should point to your local models. It seems you've already done this, but double-check that all instances of these settings are correctly configured.

2. Override Default Settings: The default settings in paperqa might still be pointing to OpenAI models. Make sure that your Settings object is correctly overriding these defaults. Here's a refined version of your setup:

   from paperqa import Settings, ask
   
   local_llm_config = {
       "model_list": [
           {
               "model_name": "ollama/llama3.2",
               "litellm_params": {
                   "model": "ollama/llama3.2",
                   "api_base": "http://localhost:11434",
               },
           }
       ]
   }
   
   settings = Settings(
       llm="ollama/llama3.2",
       llm_config=local_llm_config,
       summary_llm="ollama/llama3.2",
       summary_llm_config=local_llm_config,
       embedding="ollama/nomic-embed-text",
       paper_directory="papers",
   )
   
   answer = ask(
       "What machine learning algorithms can classify heart failure?",
       settings=settings
   )
   
   print(answer)

3. Check for Hardcoded Defaults: Ensure there are no hardcoded references to OpenAI models elsewhere in your code or configuration files. The error message suggests that a default OpenAI model might still be referenced.

4. Environment Variables: If there are any environment variables set for OpenAI API keys, ensure they are not being used. You can set a dummy value if necessary to bypass any checks:

   export OPENAI_API_KEY=foo

By following these steps, you should be able to use paperqa with only your local Ollama models and avoid any connection attempts to the OpenAI API ^[1].

To continue talking to Dosu, mention @dosu.

[floedaniel]

I have tried "everything," and nothing works. It always tries to call OPENAI_API_KEY.

[lindamathez]

it finally worked by adding agent=AgentSettings(agent_llm='ollama/llama3.2', agent_llm_config=local_llm_config) to my settings and downgrading the paperqa version from 5.5.0 to 5.3.1. Why do I need to downgrade though

[dosubot[bot]]

Downgrading from version 5.5.0 to 5.3.1 might have resolved your issue because there could be changes in version 5.5.0 that affect how local models are configured or how the AgentSettings class interacts with the rest of the system. Unfortunately, I don't have specific details on what changes were introduced in version 5.5.0 that could impact local model usage, as this information wasn't available in the repository data I accessed. It's possible that version 5.5.0 introduced stricter checks or defaults that are more aligned with OpenAI's API, which could interfere with using local models like those from Ollama.

To continue talking to Dosu, mention @dosu.

[jamesbraza]

@lindamathez can you tell me what exceptions you were getting with v5.5 that you don't with v5.3? It's hard for me to diagnose the issue without more information.

There are some Pydantic 2.10 issues being fixed in the pending v5.5.1 release that might help. Perhaps try v5.5.0 with Pydantic 2.9.

[lindamathez]

@lindamathez can you tell me what exceptions you were getting with v5.5 that you don't with v5.3? It's hard for me to diagnose the issue without more information.

There are some Pydantic 2.10 issues being fixed in the pending v5.5.1 release that might help. Perhaps try v5.5.0 with Pydantic 2.9.

It was not Pydantic, it was Encountered exception during tool call for tool complete: TypeError("Complete.complete() got an unexpected keyword argument 'ordered'"). Is there another way to solve this than downgrading?

[jamesbraza]

Hi @lindamathez , yes this is fixed by #728 on our current main branch. It will be released soon in v5.5.1.

In the meantime, you can either:

pip install git+https://github.com/Future-House/paper-qa.git
# or
pip install fhaviary<0.10.2 paper-qa>=5.5

[jamesbraza]

Okay we cut a release today under the tag v5.6.0. So just pip install -U paper-qa now and it will fix this