[Idea]: Support for the bearer token required by Crawl4AI upon connection

[garmy-ja]

What are you trying to do?

Starting with version v0.9.0, the Docker version of Crawl4AI requires a bearer token for external connections by default.
see. https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md

I would like to connect to a running Crawl4AI Docker instance from an open-notebook.

What feels difficult or missing today?

Since I am running a separate crawl4ai server alongside open-notebook, adding crawl4ai into open-notebook would result in redundancy and be a waste of resources.

What outcome would help?

By setting the bearer token for the target crawl4ai instance in an environment variable or similar, you can connect to and use the server specified by CRAWL4AI_API_URL just as before.

How do you handle this today?

No response

Additional context, examples, or possible directions

No response

How would you like to participate?

• I can add use cases or join the product/design discussion.
• I can help explore the technical approach.
• I can help implement an approved work item.
• I can test or validate a proposed solution.

Before posting

• I searched existing Discussions and Issues for related proposals.

[lfnovo]

Thanks for the clear report — and you're right. I verified the root cause: the Crawl4AI Docker client lives in content-core (the extraction library Open Notebook uses), and it currently posts to /crawl with no Authorization header at all. With Crawl4AI ≥ 0.9.0 requiring a bearer token by default, remote instances are effectively broken.

This is small and well-defined, so it graduates straight to work items — no exploration needed:

1. content-core: accept a token (env var CRAWL4AI_API_TOKEN + config key) and send it as Authorization: Bearer when configured — Crawl4AI Docker client sends no Authorization header — broken with Crawl4AI >= 0.9.0 (bearer token required) content-core#80.
2. Open Notebook: bump content-core once released, document the new variable, and pass it through the Docker setup — Support CRAWL4AI_API_TOKEN for remote Crawl4AI instances (content-core bump + docs) #1269.

Since you run a real external instance: would you be up for testing against it once the fix lands?

Status: accepted → graduated to Issues (links above).

[garmy-ja]

Thank you. I'll give it a try as much as possible!

[lfnovo]

Closing the loop: the fix is in. content-core v2.0.7 sends CRAWL4AI_API_TOKEN as Authorization: Bearer, and Open Notebook now ships with it (#1316, closing #1269): the dependency is bumped and the variable is documented in the environment reference, .env.example and docker-compose.yml. It goes out with the next release.

Since you offered to test against your remote instance: the v1-dev Docker image is rebuilt on every push to main, so you can try it before the release. Set CRAWL4AI_API_URL and CRAWL4AI_API_TOKEN in the container environment and process a JavaScript-heavy URL with the crawl4ai engine. If it still fails, comment on #1269 with the error and I'll reopen.

Status: resolved — shipped on main, released with the next version.

[garmy-ja]

Thank you!!

I have confirmed that by using the v1-dev Docker image and configuring the CRAWL4AI_API_TOKEN, I can successfully connect to Crawl4AI and retrieve the linked documents.
I also verified the connection logs on the Crawl4AI side, which confirm a successful connection.

[garmy-ja]

Thank you!!

I have confirmed that by using the v1-dev Docker image and configuring the CRAWL4AI_API_TOKEN, I can successfully connect to Crawl4AI and retrieve the linked documents.
I also verified the connection logs on the Crawl4AI side, which confirm a successful connection.

[lfnovo]

awesome.. you can come back to the main release as soon as its out in the next few days