How to skip crossref and semantic?

[sekaiyon]

By default SEMANTIC_SCHOLAR_API_KEY, CROSSREF_MAILTO and CROSSREF_API_KEY are turned on and I want to do everything offline - some of the papers and sources I use are not indexed there and I don't want to meddle with setting up the API keys (I am a noob actually). How can I turn off the check for these variables and everything they entail?

I'm getting stuff like this:

SEMANTIC_SCHOLAR_API_KEY environment variable not set. Semantic Scholar API rate limits may apply.
CROSSREF_MAILTO environment variable not set. Crossref API rate limits may apply.
CROSSREF_API_KEY environment variable not set. Crossref API rate limits may apply.
Ошибка при добавлении Illarionov_Sergei_Teoriya_poznaniya_i_filosofiya_nauki_Readli.Net_814306_original_8d958.pdf: 403, message='Forbidden', url='https://api.semanticscholar.org/graph/v1/paper/search/match?query=Example+Title&fields=authors,citationCount,citationStyles,externalIds,influentialCitationCount,isOpenAccess,journal,openAccessPdf,publicationDate,publicationTypes,title,url,venue,year'
Metadata not found for Example Title in CrossrefProvider.
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x0000023456A5EE40>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x0000023456506690>, 441312.455705)])']
connector: <aiohttp.connector.TCPConnector object at 0x0000023456A5EBA0>
SEMANTIC_SCHOLAR_API_KEY environment variable not set. Semantic Scholar API rate limits may apply.
Ошибка при добавлении Обзор литературы.pdf: 403, message='Forbidden', url='https://api.semanticscholar.org/graph/v1/paper/search/match?query=%D0%98%D1%81%D1%81%D0%BB%D0%B5%D0%B4%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5+%D1%82%D1%80%D0%B0%D0%BD%D1%81%D1%84%D0%BE%D1%80%D0%BC%D0%B0%D1%86%D0%B8%D0%B9+%D1%82%D1%80%D1%83%D0%B4%D0%B0:+%D1%82%D0%B5%D0%BE%D1%80%D0%B5%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%B8%D0%B5+%D0%BF%D0%BE%D0%B4%D1%85%D0%BE%D0%B4%D1%8B+%D0%B8+%D0%B8%D1%85+%D0%B2%D0%BB%D0%B8%D1%8F%D0%BD%D0%B8%D0%B5+%D0%BD%D0%B0+%D0%B1%D1%83%D0%B4%D1%83%D1%89%D0%B5%D0%B5+%D1%82%D1%80%D1%83%D0%B4%D0%B0&fields=authors,citationCount,citationStyles,externalIds,influentialCitationCount,isOpenAccess,journal,openAccessPdf,publicationDate,publicationTypes,title,url,venue,year'
Metadata not found for Исследование трансформаций труда: теоретические подходы и их влияние на будущее труда in CrossrefProvider.
Could not find cost for model ollama/deepseek-r1.

[sekaiyon]

I've tried various stuff and still couldn't find a way to disable checking these APIs. There is my code:

import asyncio
import os
from paperqa import Docs, Settings
from paperqa.clients import DocMetadataClient, MetadataProvider

# Заглушка для MetadataProvider
class DummyProvider(MetadataProvider):
    async def _query(self, **kwargs):
        return None  # Не делаем реальных запросов

    @property
    def query_transformer(self):
        return lambda x: x  # Возвращаем входной запрос без изменений

async def main():
    # Путь к папке с PDF
    paper_dir = "C:/Users/vasil/OneDrive/Документы/Papers"

    # Настройки: отключаем автоматическое извлечение метаданных
    settings = Settings(
        llm="ollama/deepseek-r1",
        summary_llm="ollama/deepseek-r1",
        embedding="ollama/mxbai-embed-large",
        paper_directory=paper_dir,
        parsing_use_doc_details=False,  # 🔥 Отключаем метаданные
        verbosity=3
    )

    # Создаем клиент метаданных с заглушкой
    dummy_client = DocMetadataClient(clients=[DummyProvider()])

    # Создаем Docs без вызова внешних API
    docs = Docs()

    # Меняем клиент метаданных через monkey patching
    docs._metadata_client = dummy_client  # 🔥 Обходим приватное поле

    # Добавляем PDF вручную
    pdf_files = [f for f in os.listdir(paper_dir) if f.endswith(".pdf")]
    
    for pdf_file in pdf_files:
        file_path = os.path.join(paper_dir, pdf_file)
        await docs.aadd(file_path, settings=settings, disable_clients=True)

    # Задаем вопрос
    question = "Has anyone designed neural networks that compute with proteins or DNA?"
    
    # Получаем ответ
    session = await docs.aquery(question, settings=settings)
    print("Ответ:", session.formatted_answer)

# Запуск асинхронной функции
asyncio.run(main())

The response:

SEMANTIC_SCHOLAR_API_KEY environment variable not set. Semantic Scholar API rate limits may apply.
CROSSREF_MAILTO environment variable not set. Crossref API rate limits may apply.
CROSSREF_API_KEY environment variable not set. Crossref API rate limits may apply.
Traceback (most recent call last):
  File "C:\Users\vasil\OneDrive\Документы\Papers\test-ollama.py", line 53, in <module>
    asyncio.run(main())
    ~~~~~~~~~~~^^^^^^^^
  File "C:\Program Files\Lib\asyncio\runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "C:\Program Files\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "C:\Program Files\Lib\asyncio\base_events.py", line 719, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "C:\Users\vasil\OneDrive\Документы\Papers\test-ollama.py", line 43, in main
    await docs.aadd(file_path, settings=settings, disable_clients=True)
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\paperqa\docs.py", line 375, in aadd
    doc = await metadata_client.upgrade_doc_to_doc_details(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        doc, **(query_kwargs | kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\paperqa\clients\__init__.py", line 212, in upgrade_doc_to_doc_details
    if doc_details := await self.query(**kwargs):
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\paperqa\clients\__init__.py", line 153, in query
    await gather_with_concurrency(
        len(task.providers), task.provider_queries(query_args)
    )
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\lmi\utils.py", line 100, in gather_with_concurrency
    return await asyncio.gather(*(sem_coro(c) for c in coros))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\lmi\utils.py", line 87, in sem_coro
    return await coro
           ^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\paperqa\clients\client_models.py", line 110, in query
    return await self._query(client_query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\paperqa\clients\semantic_scholar.py", line 350, in _query
    return await get_s2_doc_details_from_title(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\paperqa\clients\semantic_scholar.py", line 335, in get_s2_doc_details_from_title
    return await s2_title_search(
           ^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\paperqa\clients\semantic_scholar.py", line 238, in s2_title_search
    data = await _s2_get_with_retrying(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\asyncio\__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\asyncio\__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\asyncio\__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\_utils.py", line 99, in inner
    return call(*args, **kwargs)
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\__init__.py", line 400, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ~~~~~~~~~~~~~~~~~^^
  File "C:\Program Files\Lib\concurrent\futures\_base.py", line 449, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "C:\Program Files\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\asyncio\__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\paperqa\clients\semantic_scholar.py", line 119, in _s2_get_with_retrying
    return await _get_with_retrying(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\asyncio\__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\asyncio\__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\asyncio\__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\_utils.py", line 99, in inner
    return call(*args, **kwargs)
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\__init__.py", line 400, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ~~~~~~~~~~~~~~~~~^^
  File "C:\Program Files\Lib\concurrent\futures\_base.py", line 449, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "C:\Program Files\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\tenacity\asyncio\__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\paperqa\utils.py", line 447, in _get_with_retrying
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\vasil\AppData\Roaming\Python\Python313\site-packages\aiohttp\client_reqrep.py", line 1243, in raise_for_status
    raise ClientResponseError(
    ...<5 lines>...
    )
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url='https://api.semanticscholar.org/graph/v1/paper/search/match?query=Example+Title&fields=authors,citationCount,citationStyles,externalIds,influentialCitationCount,isOpenAccess,journal,openAccessPdf,publicationDate,publicationTypes,title,url,venue,year'
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x0000019753C470E0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x00000197536F2870>, 457557.9897172)])']
connector: <aiohttp.connector.TCPConnector object at 0x0000019753C46E40>

What shall I do? How can I fix this if using these API-s is a no-go?

[sekaiyon]

@dosu

[jamesbraza]

Hi @sekaiyon it seems you have stumped Dosu :), Dosu only answers when confident.

As a workaround, just go to paperqa.clients.__init__.py and comment out SemanticScholarProvider from the DEFAULT_CLIENTS: https://github.com/Future-House/paper-qa/blob/v5.20.0/paperqa/clients/__init__.py#L26

For a real solution, I will answer tomorrow.

[dosubot]

To fully disable all calls to Semantic Scholar and Crossref APIs in paper-qa and work offline, you need to ensure that no code path attempts to use those providers. Your approach with a DummyProvider is on the right track, but the errors show that the default providers are still being used somewhere in the process.

Here’s what’s happening and how to fix it:

Why your current approach isn’t working

• Even with disable_clients=True and monkey-patching, the default metadata providers (SemanticScholarProvider, CrossrefProvider) are still being called internally when you add documents.
• The warnings and errors about missing API keys and 403s confirm that the code is still trying to reach out to those APIs, just without credentials, which leads to failures.

How to actually disable all API calls

You must ensure that only your DummyProvider is ever used for metadata.
Here’s how you can do it:

1. Explicitly pass your dummy metadata client everywhere.
   When you create your Docs object, make sure it is initialized with your dummy client and that no other providers are registered.

2. Avoid any code path that might re-initialize or override the metadata client.
   If you use docs.aadd(...), make sure it does not internally create a new metadata client with default providers.

3. Set parsing_use_doc_details=False in your Settings (which you already do), so that metadata extraction is not attempted.

4. Double-check that your DummyProvider is the only provider.
   If you see any warnings or errors about missing API keys, it means the default providers are still being used somewhere.

Example: Forcing Only DummyProvider

from paperqa import Docs, Settings
from paperqa.clients import DocMetadataClient, MetadataProvider

class DummyProvider(MetadataProvider):
    async def _query(self, **kwargs):
        return None
    @property
    def query_transformer(self):
        return lambda x: x

dummy_client = DocMetadataClient(clients=[DummyProvider()])

# Pass the dummy_client directly to Docs (if supported by your version)
docs = Docs(metadata_client=dummy_client)

# Or, after creation, forcibly override the private field
docs._metadata_client = dummy_client

# Now add documents as before

If the Docs constructor does not accept a metadata_client argument, your monkey-patching approach is correct, but do it immediately after creating the Docs instance and before adding any documents.

If you still see API calls

• Double-check that you are not calling any function that might internally create a new metadata client.
• Make sure you are not using any CLI tools or scripts that ignore your monkey-patching.
• If you are using a recent version of paper-qa and this still doesn’t work, it may be a limitation or bug in the library’s design.

Summary

• Your DummyProvider approach is correct, but you must ensure it is the only provider used.
• If you see any API warnings or errors, the default providers are still being used somewhere.
• There is no documented setting to globally disable all API usage; you must override the metadata client as above [source] [source].

If you need a more concrete code example tailored to your exact version of paper-qa, let me know which version you are using.

To continue talking to Dosu, mention @dosu-bot.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[jamesbraza]

Nevermind you didn't stump Dosu, just it was a bit slow today 😆

[sekaiyon]

Some updates.

I've backrolled my code a bit to get rid of some mistakes and looked up some stuff.

import asyncio
import os
from paperqa import Docs, Settings

async def main():
    # Путь к папке с PDF
    paper_dir = "C:/Users/vasil/OneDrive/Документы/Papers"

    # Настройки: отключаем метаданные и используем Ollama + DeepSeek R1
    settings = Settings(
        llm="ollama/deepseek-r1",
        summary_llm="ollama/deepseek-r1",
        embedding="ollama/mxbai-embed-large",
        paper_directory=paper_dir,
        parsing_use_doc_details=False,

    )

    docs = Docs()

    # Добавляем PDF вручную
    pdf_files = [f for f in os.listdir(paper_dir) if f.endswith(".pdf")]
    for pdf_file in pdf_files:
        file_path = os.path.join(paper_dir, pdf_file)
        try:
            await docs.aadd(file_path, settings=settings, disable_clients=True)  # 🔥 Отключаем внешние запросы
        except Exception as e:
            print(f"Ошибка при добавлении {pdf_file}: {str(e)}")

    # Задаем вопрос
    question = "Has anyone designed neural networks that compute with proteins or DNA?"
    
    # Получаем ответ
    try:
        session = await docs.aquery(question, settings=settings)
        print("Ответ:", session.formatted_answer)
    except Exception as e:
        print("Ошибка при ответе:", str(e))

# Запуск асинхронной функции
asyncio.run(main())
Setting false in both of these files doesn't make a difference ([it helped the guy](https://github.com/Future-House/paper-qa/issues/547))

Commenting on the variables does turn them off, but still, I get this:

Ошибка при добавлении Illarionov_Sergei_Teoriya_poznaniya_i_filosofiya_nauki_Readli.Net_814306_original_8d958.pdf: At least one MetadataProvider must be provided.
Ошибка при добавлении Обзор литературы.pdf: At least one MetadataProvider must be provided.
Could not find cost for model ollama/deepseek-r1.
Ответ: Question: Has anyone designed neural networks that compute with proteins or DNA?

<think>
Okay, I need to figure out if anyone has designed neural networks that compute with proteins or DNA based on the given context. Let me start by looking through the provided context snippets.

First, I see a mention of "Protein-based computing" in the context (2015). It says that researchers are exploring this area but it's still in its early stages and not yet practical for real-world applications. That suggests there have been attempts to create neural networks using proteins, though they're not widely used yet.

Next, looking at DNA computing, I find a reference from 2018 about "DNA-based neural networks." It mentions that while research is ongoing, it's still in the experimental phase and hasn't achieved widespread adoption. This indicates that there have been efforts to integrate DNA into computational models, but they're not yet practical.

Additionally, another context snippet talks about "Membrane computing" (2016), which uses biological structures like cells or organelles as a basis for computation. It's described as a promising field with potential applications in areas where traditional computing isn't suitable. This supports the idea that there are alternative computational models inspired by biology, including proteins and DNA.

Putting this together, it seems that while neural networks using proteins or DNA haven't become mainstream, there have been experimental designs and research efforts in these areas. The key points are that they're still in development and not yet practical for everyday use.
</think>

As of current research, neural networks designed to compute with proteins or DNA are still largely experimental and theoretical constructs. Here's a concise overview:

1. **Protein-based Computing**: Researchers have explored the idea of creating neural networks using proteins (2015). These efforts are in the early stages and primarily serve as proof-of-concept experiments, not yet practical for real-world applications.

2. **DNA-based Neural Networks**: Although less advanced than protein-based systems, DNA-based neural networks have been studied experimentally (2018). These models are still in development and lack widespread adoption.

3. **Membrane Computing**: This computational model, inspired by biological structures like cells or organelles, represents a promising alternative approach to traditional computing (2016). It offers potential applications where conventional methods may be inadequate but is not yet fully integrated into practical neural network designs involving proteins or DNA.

In summary, while there have been experimental attempts to design neural networks using proteins and DNA, these approaches remain in the developmental stage and are not yet practical for widespread use.

How do I make that he looks into the files and bypasses the Metadata check?

Sorry, I am really bad at it, I don't know how to do better than for i in range() x)