Internal Service β This is not an HTTP endpoint. Itβs called directly by the
generate-all orchestrator.Purpose
Uses the Firecrawl Agent API to extract what a business does. This information is used to generate the AI website content (llms.txt, Q&A pages, data.json). Runs in GROUP 1a (parallel with all other GROUP 1 tasks).Function Signature
Parameters
Returns
How It Works
Firecrawl Agent Prompt
The service sends this prompt to Firecrawl Agent:Schema
Used By
This is separate from the Scrape Website service (GROUP 1b). Scraped pages are only used for markdown replica generation, not for LLM content.
Why Separate from Scraping?
Benefits:
- Focused prompts: LLM content generation gets clean, structured business info
- No token waste: Donβt send full page markdown to Gemini for llms.txt generation
- Parallel execution: Both can run simultaneously since theyβre independent
Code Location
Error Handling
- Timeout - Firecrawl agent takes too long (max 3 minutes)
- API error - Firecrawl API returns non-200 status
- Missing API key -
FIRECRAWL_API_KEYnot configured