feat: add scrapegraph tool #470
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
This is the final PR Bugbot will review for you during this billing cycle
Your free Bugbot reviews will reset on October 28
Details
Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
| self._validate_url(website_url) | ||
| self.website_url = website_url | ||
| self.description = f"A tool that uses Scrapegraph AI to intelligently scrape {website_url}'s content." | ||
| self.description = f"A tool that uses ScrapeGraph AI to scrape {website_url}'s content using {method.value} method." |
There was a problem hiding this comment.
Bug: Constructor Parameter Ignored in Tool Method
The method parameter in the ScrapegraphScrapeTool constructor is used to set the tool's description but isn't stored as an instance variable. The _run method then determines the actual scraping method from its kwargs or defaults to SMARTSCRAPER, ignoring the method specified during initialization. This can lead to the tool's description misrepresenting the method actually used.
| status = result.get("status") | ||
|
|
||
| if status == "success" and result.get("result"): | ||
| return result["result"].get("llm_result", result["result"]) |
There was a problem hiding this comment.
Bug: Async Crawl Fails on Non-Dictionary Result
The _handle_async_crawl method attempts to call .get("llm_result") on result["result"], assuming it's always a dictionary. If the API returns result["result"] as a non-dictionary type (like a string or list), this will cause an AttributeError.
1 participant
Note
Refactors the Scrapegraph tool into a multi-method ScrapeGraph scraper with enum-driven methods, richer schemas/validation, async crawl handling, expanded docs, tests, and dependency bump.
ScrapeMethodenum withsmartscraper,searchscraper,agenticscraper,crawl,scrape,markdownify.FixedScrapegraphScrapeToolSchemaand extendsScrapegraphScrapeToolSchemawith fields/validators (num_results,depth,timeout).tools/__init__.pyandscrapegraph_scrape_tool/__init__.py.README.mdwith features, setup, advanced examples, error handling, and API reference.scrapegraph-pyto>=1.31.0.tests/tools/test_scrapegraph_scrape_tool.py.Written by Cursor Bugbot for commit e96d426. This will update automatically on new commits. Configure here.