feat: add FIRECRAWL_API_BASE_URL environment variable support #171
Conversation
Allow users to configure a custom Firecrawl API base URL via the FIRECRAWL_API_BASE_URL environment variable. This is useful for: - Self-hosted Firecrawl instances - Custom proxy endpoints - Development/testing environments Defaults to https://api.firecrawl.dev when not set.
Document the new FIRECRAWL_API_BASE_URL environment variable in the example configuration file.
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for configurable Firecrawl API base URLs, enabling the use of self-hosted Firecrawl instances instead of hardcoding the production API endpoint.
- Introduced
FIRECRAWL_API_BASE_URLenvironment variable with fallback to the default production URL - Updated
.env.examplewith documentation for the new configuration option
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| productionized/pulse-fetch/shared/src/scraping-client/lib/firecrawl-scrape.ts | Replaced hardcoded Firecrawl API URL with configurable base URL from environment variable |
| productionized/pulse-fetch/.env.example | Added documentation and example configuration for the new FIRECRAWL_API_BASE_URL environment variable |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| }> { | ||
| try { | ||
| const response = await fetch('https://api.firecrawl.dev/v1/scrape', { | ||
| const baseUrl = process.env.FIRECRAWL_API_BASE_URL || 'https://api.firecrawl.dev'; |
There was a problem hiding this comment.
The base URL should be validated to prevent injection attacks. Consider validating that it starts with 'http://' or 'https://' and doesn't contain path traversal characters. For example: if (baseUrl && !/^https?:\/\/.+$/.test(baseUrl)) { throw new Error('Invalid FIRECRAWL_API_BASE_URL'); }
| const baseUrl = process.env.FIRECRAWL_API_BASE_URL || 'https://api.firecrawl.dev'; | |
| const baseUrl = process.env.FIRECRAWL_API_BASE_URL || 'https://api.firecrawl.dev'; | |
| // Validate baseUrl to prevent injection attacks | |
| if ( | |
| baseUrl && | |
| (!/^https?:\/\/[^\\]+$/.test(baseUrl) || baseUrl.includes('..')) | |
| ) { | |
| throw new Error('Invalid FIRECRAWL_API_BASE_URL'); | |
| } |
| }> { | ||
| try { | ||
| const response = await fetch('https://api.firecrawl.dev/v1/scrape', { | ||
| const baseUrl = process.env.FIRECRAWL_API_BASE_URL || 'https://api.firecrawl.dev'; |
There was a problem hiding this comment.
The base URL is resolved on every function call. Consider moving this to a module-level constant to avoid repeated environment variable lookups and improve performance.
- Add validation to prevent URL injection attacks - Move baseUrl resolution to module-level constant for performance - Validate that URL starts with http:// or https:// - Prevent path traversal attacks Co-authored-by: GitHub Copilot
2 participants
Description
This PR adds support for configuring a custom Firecrawl API base URL via the
FIRECRAWL_API_BASE_URLenvironment variable.Motivation
Currently, the Firecrawl API URL is hardcoded to
https://api.firecrawl.dev/v1/scrape. This prevents users from:Changes
Modified
firecrawl-scrape.ts: Updated thescrapeWithFirecrawlfunction to read the base URL fromprocess.env.FIRECRAWL_API_BASE_URL, defaulting tohttps://api.firecrawl.devwhen not set.Updated
.env.example: Added documentation for the newFIRECRAWL_API_BASE_URLenvironment variable.Usage
Users can now set a custom Firecrawl API base URL:
# .env FIRECRAWL_API_BASE_URL=https://my-custom-firecrawl.example.comIf not set, the default (
https://api.firecrawl.dev) is used, maintaining backward compatibility.Testing