​

Internal Service: scrape_ai_site

​

Function Signature

async def scrape_ai_site(deployment_url: str, product_name: str = None) -> dict

​

Parameters

​

Returns

# Success
{
    "status": "success",
    "llms_content": "...",  # Root llms.txt (business context)
    "product_llms_content": "..." or None,  # Product-specific llms.txt
    "ai_site_url": "https://..."  # Base URL (cleaned)
}

# Error
{
    "status": "error",
    "error": "Could not fetch llms.txt from AI site"
}

​

Scalable Product LLMs Architecture

​

For Business AI Articles:

• Fetches only /llms.txt
• Uses general business context

​

For Product AI Articles:

• Fetches /llms.txt (business context)
• Also fetches /llms/{product-slug}.txt (product details)
• Falls back to root llms.txt if product file doesn’t exist

​

Behavior

1. Strips trailing slash from deployment URL
2. Adds Vercel bypass protection header if VERCEL_BYPASS_PROTECTION_SECRET is set
3. Fetches /llms.txt with up to 5 retries (8 second delay between retries)
4. If product_name is provided, also fetches /llms/{product-slug}.txt
5. Returns the content for use in Step 3

​

Why This Architecture?

• Constant memory - Each AI article loads ~5KB total
• No token limits - Never hits Gemini’s context window
• Fast - Small files = fast fetches
• Independent - Update one product without touching others

​

Environment Variables

​

Code Location

src/app/apis/cron/create_boosted_page/children/step_2_scrape_ai_site.py