Integrate crawl4ai tool

[rjambrecic]

Why are these changes needed?

We want to be able to use crawl4ai framework:

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]

llm_config = {
    "config_list": config_list,
}

user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
assistant = AssistantAgent(name="assistant", llm_config=llm_config)

class Blog(BaseModel):
    title: str
    url: str


# Set llm_config and extraction_model to Crawl4AITool
crawlai_tool = Crawl4AITool(llm_config=llm_config, extraction_model=Blog)

crawlai_tool.register_for_execution(user_proxy)
crawlai_tool.register_for_llm(assistant)

message = "Extract all blog posts from https://docs.ag2.ai/blog"
result = user_proxy.initiate_chat(
    recipient=assistant,
    message=message,
    max_turns=2,
)

user_proxy (to assistant):

Extract all blog posts from https://docs.ag2.ai/blog

--------------------------------------------------------------------------------
assistant (to user_proxy):

***** Suggested tool call (call_dCeqbTvmUZ79ZIQyeYXlleEq): crawl4ai *****
Arguments: 
{"url":"https://docs.ag2.ai/blog","instruction":"Extract all blog post titles and links."}
*************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION crawl4ai...
Call ID: call_dCeqbTvmUZ79ZIQyeYXlleEq
Input arguments: {'url': 'https://docs.ag2.ai/blog', 'instruction': 'Extract all blog post titles and links.'}
[INIT].... → Crawl4AI 0.4.247
[FETCH]... ↓ https://docs.ag2.ai/blog... | Status: True | Time: 1.89s
[SCRAPE].. ◆ Processed https://docs.ag2.ai/blog... | Time: 55ms
INFO:httpx:HTTP Request: GET https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json "HTTP/1.1 200 OK"
[/Users/robert/projects/ag2/.venv-crawl4ai/lib/python3.11/site-packages/pydantic/_internal/_config.py:345](https://file+.vscode-resource.vscode-cdn.net/Users/robert/projects/ag2/.venv-crawl4ai/lib/python3.11/site-packages/pydantic/_internal/_config.py:345): UserWarning: Valid config keys have changed in V2:
* 'fields' has been removed
  warnings.warn(message, UserWarning)
12:42:49 - LiteLLM:INFO: utils.py:2825 - 
LiteLLM completion() model= gpt-4o-mini; provider = openai
INFO:LiteLLM:
LiteLLM completion() model= gpt-4o-mini; provider = openai
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
12:43:18 - LiteLLM:INFO: utils.py:1030 - Wrapper: Completed Call, calling success_handler
INFO:LiteLLM:Wrapper: Completed Call, calling success_handler
[EXTRACT]. ■ Completed for https://docs.ag2.ai/blog... | Time: 29.94337983400328s
[COMPLETE] ● https://docs.ag2.ai/blog... | Status: True | Total: 31.90s
user_proxy (to assistant):

***** Response from calling tool (call_dCeqbTvmUZ79ZIQyeYXlleEq) *****
[
    {
        "title": "Tools with ChatContext Dependency Injection",
        "url": "https://docs.ag2.ai/blog/2025-01-22-Tools-ChatContext-Dependency-Injection/index",
        "error": false
    },
    {
        "title": "Streaming input and output using WebSockets",
        "url": "https://docs.ag2.ai/blog/2025-01-10-WebSockets/index",
        "error": false
    },
    {
        "title": "Real-Time Voice Interactions over WebRTC",
        "url": "https://docs.ag2.ai/blog/2025-01-09-RealtimeAgent-over-WebRTC/index",
        "error": false
    },
    {
        "title": "Real-Time Voice Interactions with the WebSocket Audio Adapter",
        "url": "https://docs.ag2.ai/blog/2025-01-08-RealtimeAgent-over-websocket/index",
        "error": false
    },
    {
        "title": "Tools Dependency Injection",
        "url": "https://docs.ag2.ai/blog/2025-01-07-Tools-Dependency-Injection/index",
        "error": false
    },
    {
        "title": "Cross-Framework LLM Tool Integration with AG2",
        "url": "https://docs.ag2.ai/blog/2024-12-20-Tools-interoperability/index",
        "error": false
    },
    {
        "title": "ReasoningAgent Update - Beam Search, MCTS, and LATS for LLM Reasoning",
        "url": "https://docs.ag2.ai/blog/2024-12-20-Reasoning-Update/index",
        "error": false
    },
    {
        "title": "Introducing RealtimeAgent Capabilities in AG2",
        "url": "https://docs.ag2.ai/blog/2024-12-20-RealtimeAgent/index",
        "error": false
    },
    ..........
    }
]
**********************************************************************

--------------------------------------------------------------------------------
assistant (to user_proxy):

I have extracted all the blog posts from the specified website. Here are the titles and links:

1. [Tools with ChatContext Dependency Injection](https://docs.ag2.ai/blog/2025-01-22-Tools-ChatContext-Dependency-Injection/index)
2. [Streaming input and output using WebSockets](https://docs.ag2.ai/blog/2025-01-10-WebSockets/index)
3. [Real-Time Voice Interactions over WebRTC](https://docs.ag2.ai/blog/2025-01-09-RealtimeAgent-over-WebRTC/index)
4. [Real-Time Voice Interactions with the WebSocket Audio Adapter](https://docs.ag2.ai/blog/2025-01-08-RealtimeAgent-over-websocket/index)
5. [Tools Dependency Injection](https://docs.ag2.ai/blog/2025-01-07-Tools-Dependency-Injection/index)
6. [Cross-Framework LLM Tool Integration with AG2](https://docs.ag2.ai/blog/2024-12-20-Tools-interoperability/index)
7. [ReasoningAgent Update - Beam Search, MCTS, and LATS for LLM Reasoning](https://docs.ag2.ai/blog/2024-12-20-Reasoning-Update/index)
8. [Introducing RealtimeAgent Capabilities in AG2](https://docs.ag2.ai/blog/2024-12-20-RealtimeAgent/index)
....

If you need further information or have specific tasks related to these blog posts, let me know!

TERMINATE

Related issue number

Closes #658

Checks

• I've included any doc changes needed for https://docs.ag2.ai/. See https://docs.ag2.ai/docs/contributor-guide/documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.