filtering crawling links using llms

[kashyab12]

Hey! Love crawlee so far. The main issue I am facing is I want to filtering the urls to crawl for a given page using LLMs. Is there a clean way to do this? So far I implemented a transformer for enqueue_links which saves the links to a dict and then process those dicts at a later point of time using another crawler object. Any other suggestions to solve this problem? I don't want to make the llm call in the transform function since that would be an LLM call per URL found which is quite expensive.

[janbuchar]

Hello, so do I understand it right that you want to store the links you find on a page and only filter and enqueue them after you gather a bigger batch?

[vdusek]

Hi, once we merge PR #1024, it should help with your use case. You'll be able to call just extract_links (without enqueueing them), filter the results as needed, store the links somewhere, and then enqueue them later using add_requests(extracted_links). I believe that's what you need.