Firecrawl - Extract actions #16069
Firecrawl - Extract actions #16069
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 3 Skipped Deployments
|
WalkthroughThis PR updates several Firecrawl modules by incrementing version numbers and adding new functionality. Three existing actions (crawl-url, get-crawl-status, scrape-page) have their version properties incremented, while two new actions—extract-data and get-extract-status—are introduced for data extraction and status retrieval. Additionally, the Firecrawl app now includes methods to initiate extraction and check its status. The package version in package.json is also updated. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant ExtractAction as "firecrawl-extract-data"
participant App as "Firecrawl App"
participant API as "Firecrawl API"
Client->>ExtractAction: Trigger extraction with options (urls, prompt/schema)
ExtractAction->>App: Call extract(opts)
App->>API: POST /extract with opts
API-->>App: Extraction job initiated
App-->>ExtractAction: Return job ID & summary
ExtractAction-->>Client: Provide extraction response
sequenceDiagram
participant ExtractAction as "firecrawl-extract-data"
participant API as "Firecrawl API"
ExtractAction->>API: GET /extract/{id} (poll every 3 sec)
loop While status is "processing"
API-->>ExtractAction: Return "processing" status
ExtractAction->>API: Poll again
end
API-->>ExtractAction: Final status update
ExtractAction-->>Client: Return completed status
Assessment against linked issues
Suggested labels
Suggested reviewers
Poem
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
components/bloomerang/bloomerang.app.mjsOops! Something went wrong! :( ESLint: 8.57.1 Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs components/firecrawl/actions/crawl-url/crawl-url.mjsOops! Something went wrong! :( ESLint: 8.57.1 Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs components/firecrawl/actions/extract-data/extract-data.mjsOops! Something went wrong! :( ESLint: 8.57.1 Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs
✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
components/firecrawl/firecrawl.app.mjs (1)
72-79: Consider consistent parameter naming for ID properties.The method looks well-implemented, but there's a minor inconsistency in parameter naming. The
getCrawlStatusmethod usescrawlIdwhile this method usesid. For better consistency, consider usingextractIdinstead.getExtractStatus({ - id, ...opts + extractId, ...opts }) { return this._makeRequest({ - path: `/extract/${id}`, + path: `/extract/${extractId}`, ...opts, }); },components/firecrawl/actions/get-extract-status/get-extract-status.mjs (1)
17-25: Add error handling to the run methodThe current implementation doesn't handle any errors that might occur during the API call. Consider adding try/catch block to provide a better user experience with meaningful error messages.
async run({ $ }) { + try { const response = await this.firecrawl.getExtractStatus({ $, id: this.extractId, }); $.export("$summary", `Successfully retrieved status for extract (ID: ${this.extractId})`); return response; + } catch (error) { + $.export("$summary", `Failed to retrieve status for extract (ID: ${this.extractId})`); + throw error; + } },components/firecrawl/actions/extract-data/extract-data.mjs (2)
37-40: Clarify the description for importSitemap propertyThe current description "When true, sitemap.xml files will be ignored during website scanning" is counter-intuitive. If "importSitemap" is true, one would expect sitemaps to be included, not ignored. Please verify if this is correct or update the description.
importSitemap: { type: "boolean", label: "Import Sitemap", - description: "When true, sitemap.xml files will be ignored during website scanning", + description: "When true, sitemap.xml files will be imported and used during website scanning", optional: true, },
91-93: Handle unsuccessful extraction scenariosThe success message is only exported if response.success is true, but there's no handling for failed extractions. Consider adding a condition to export an appropriate message when the extraction is not successful.
if (response.success) { $.export("$summary", "Successfully extracted data."); + } else { + $.export("$summary", `Extraction completed but was not successful: ${response.error || 'Unknown error'}`); }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
components/firecrawl/actions/crawl-url/crawl-url.mjs(1 hunks)components/firecrawl/actions/extract-data/extract-data.mjs(1 hunks)components/firecrawl/actions/get-crawl-status/get-crawl-status.mjs(1 hunks)components/firecrawl/actions/get-extract-status/get-extract-status.mjs(1 hunks)components/firecrawl/actions/scrape-page/scrape-page.mjs(1 hunks)components/firecrawl/firecrawl.app.mjs(1 hunks)components/firecrawl/package.json(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: Verify TypeScript components
- GitHub Check: pnpm publish
- GitHub Check: Publish TypeScript components
🔇 Additional comments (6)
components/firecrawl/package.json (1)
3-3: Version bump to 1.1.0 looks appropriate.The version increment from 1.0.0 to 1.1.0 aligns with the addition of new extract-related functionality in this update.
components/firecrawl/actions/crawl-url/crawl-url.mjs (1)
8-8: Version increment looks good.Patch version update from 1.0.0 to 1.0.1 is appropriate for maintaining version consistency across components.
components/firecrawl/actions/scrape-page/scrape-page.mjs (1)
10-10: Version increment looks good.Patch version update from 1.0.0 to 1.0.1 is appropriate for maintaining version consistency across components.
components/firecrawl/actions/get-crawl-status/get-crawl-status.mjs (1)
7-7: Version increment looks good.Patch version update from 0.0.2 to 0.0.3 is appropriate for maintaining version consistency across components.
components/firecrawl/firecrawl.app.mjs (1)
65-71: Added extract method looks good.The implementation follows the established pattern of other API methods in this file, properly utilizing the
_makeRequesthelper function.components/firecrawl/actions/extract-data/extract-data.mjs (1)
62-64:✅ Verification successful
Enhance input validation
The validation ensures either prompt or schema is provided, but doesn't check if both are provided. If having both is valid, this is fine, but it might be worth considering more precise validation if only one should be used at a time.
🌐 Web query:
Can Firecrawl API use both prompt and schema parameters simultaneously?💡 Result:
Yes, Firecrawl API allows simultaneous use of both
promptandschemaparameters to enhance structured data extraction. This combination enables users to guide the AI's output structure while providing contextual instructions:Key Implementation Details
Schema-Driven Structure
Theschemaparameter defines the required output format using Pydantic models or JSON Schema, ensuring consistent data structure and type validation[15][14][3].Prompt-Based Guidance
Thepromptparameter provides natural language instructions to help the LLM understand extraction priorities and contextual nuances[7][15][3].Combined Usage Example
from firecrawl import FirecrawlApp from pydantic import BaseModel, Field class ProductSchema(BaseModel): name: str = Field(description="Product name from header") price: float = Field(description="Current price in USD") features: list[str] = Field(description="Key product features") app = FirecrawlApp(api_key="YOUR_KEY") data = app.scrape_url( 'https://example-store.com/product123', params={ 'jsonOptions': { 'schema': ProductSchema.model_json_schema(), 'prompt': "Focus on technical specs and ignore marketing fluff" } } )Benefits of Dual Parameter Use
Structured Output Control
Schema enforces strict typing and nested data formats[14][15]Contextual Flexibility
Prompts clarify ambiguous cases within the schema constraints[3][7]Enhanced Accuracy
Combination reduces LLM hallucination risks by 37% compared to prompt-only extractions[14]This hybrid approach is particularly useful for complex extraction tasks requiring both strict formatting and human-readable instructions[15][3]. The schema handles data validation while the prompt guides content interpretation, forming a robust extraction pipeline.
Citations:
- 1: https://docs.firecrawl.dev/api-reference/endpoint/crawl-post
- 2: https://docs.firecrawl.dev/features/crawl
- 3: https://docs.firecrawl.dev/advanced-scraping-guide
- 4: https://docs.firecrawl.dev/v1-welcome
- 5: https://js.langchain.com/docs/integrations/document_loaders/web_loaders/firecrawl
- 6: https://glama.ai/mcp/servers/@mendableai/firecrawl-mcp-server
- 7: https://docs.firecrawl.dev/introduction
- 8: https://www.firecrawl.dev
- 9: https://www.youtube.com/watch?v=qk1iQKHaKiI
- 10: https://github.com/devflowinc/firecrawl-simple/blob/main/apps/api/v1-openapi.json
- 11: https://github.com/mendableai/firecrawl
- 12: https://glama.ai/mcp/servers/@Msparihar/mcp-server-firecrawl
- 13: https://docs.firecrawl.dev/api-reference/endpoint/scrape
- 14: https://www.firecrawl.dev/blog/mastering-firecrawl-scrape-endpoint
- 15: https://www.firecrawl.dev/blog/mastering-firecrawl-extract-endpoint
- 16: https://docs.firecrawl.dev/features/extract
- 17: https://github.com/mendableai/firecrawl/blob/main/README.md
- 18: https://glama.ai/mcp/servers/@codyde/mcp-firecrawl-tool
Input Validation is Correct
The current validation in
components/firecrawl/actions/extract-data/extract-data.mjs(lines 62-64) correctly enforces that at least one ofpromptorschemais provided. Given that the Firecrawl API supports using both parameters simultaneously to enhance structured extraction, no additional validation changes are necessary.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
components/firecrawl/actions/extract-data/extract-data.mjs (1)
79-89:⚠️ Potential issueImplement timeout for polling mechanism
The current polling mechanism doesn't have a timeout or maximum retry count, which could lead to the action running indefinitely if the status never changes from "processing". Consider adding a timeout mechanism.
if (this.waitForCompletion) { const id = response.id; const timer = (ms) => new Promise((res) => setTimeout(res, ms)); + const MAX_RETRIES = 60; // 3 minutes (60 * 3 seconds) + let retries = 0; do { response = await this.firecrawl.getExtractStatus({ $, id, }); await timer(3000); + retries++; + if (retries >= MAX_RETRIES) { + $.export("$summary", "Extraction is taking longer than expected. You can check the status later using the Get Extract Status action."); + break; + } } while (response.status === "processing"); }
🧹 Nitpick comments (3)
components/firecrawl/actions/extract-data/extract-data.mjs (3)
24-29: Consider adding schema validation or documentationThe schema property is defined as type "object" but there's no indication of what structure users should follow. Consider either adding validation for the schema format or enhancing the description with an example of the expected structure to guide users.
91-93: Enhance response summary with more detailsThe current success summary is generic and doesn't provide much information about what was extracted. Consider enhancing it with details about the extraction, such as the number of URLs processed or the type of data extracted.
if (response.success) { - $.export("$summary", "Successfully extracted data."); + const urlCount = this.urls.length; + $.export("$summary", `Successfully extracted data from ${urlCount} URL${urlCount !== 1 ? 's' : ''}.`); + } else { + $.export("$summary", `Extraction completed with status: ${response.status || 'unknown'}`); }
71-71: Consider validating schema structure before sending to APIThe current implementation passes the schema directly to the API after parsing. It might be beneficial to add validation to ensure the schema has the required structure before making the API call.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
components/bloomerang/bloomerang.app.mjs(1 hunks)components/firecrawl/actions/crawl-url/crawl-url.mjs(1 hunks)components/firecrawl/actions/extract-data/extract-data.mjs(1 hunks)components/hyperbrowser/hyperbrowser.app.mjs(1 hunks)components/nutrient_workflow_automation/nutrient_workflow_automation.app.mjs(1 hunks)components/oracle_cloud_infrastructure/oracle_cloud_infrastructure.app.mjs(1 hunks)
✅ Files skipped from review due to trivial changes (4)
- components/oracle_cloud_infrastructure/oracle_cloud_infrastructure.app.mjs
- components/bloomerang/bloomerang.app.mjs
- components/nutrient_workflow_automation/nutrient_workflow_automation.app.mjs
- components/hyperbrowser/hyperbrowser.app.mjs
🚧 Files skipped from review as they are similar to previous changes (1)
- components/firecrawl/actions/crawl-url/crawl-url.mjs
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: Verify TypeScript components
- GitHub Check: pnpm publish
- GitHub Check: Publish TypeScript components
|
/approve |
Resolves #16022
Summary by CodeRabbit
New Features
Chores