Detect Changes

1. Custom Mapper → Get current URL list (sitemap + robots.txt + HTML links) [FREE]
2. Compare URLs vs stored site_map
   - New URLs = new pages
   - Missing URLs = removed pages
3. Hashing Service → Fetch raw HTML + hash for existing pages [FREE]
4. Compare hashes vs stored hashes
   - Hash mismatch = content changed
5. Batch Scrape ONLY new + changed URLs → Get markdown [PAID - only what's needed]
6. Return all data for downstream APIs

{
  "status": "success",  // or "unchanged" or "error"
  "new_pages": [
    {"url": "https://example.com/new-page", "markdown": "...", "hash": "abc123"}
  ],
  "changed_pages": [
    {"url": "https://example.com/about", "markdown": "...", "old_hash": "def456", "new_hash": "ghi789"}
  ],
  "removed_urls": ["https://example.com/old-page"],
  "unchanged_pages": [
    {"url": "https://example.com/", "hash": "..."}
  ],
  "updated_site_map": ["https://example.com/", "https://example.com/about", ...],
  "updated_hashes": {"https://example.com/": "abc123", ...},
  "business_info": {
    "entity_id": "uuid",
    "url": "https://example.com",
    "name": "Example Company",
    "clerk_org_id": "org_xxx",
    "ai_site_id": "uuid",
    "deployment_url": "https://example.searchcompany.dev",
    "project_name": "example-searchcompany-dev"
  }
}

# Batch 1a: Detect changes and update AI site
changes = await detect_changes(business_id)

if changes["status"] == "unchanged":
    return  # Nothing to do for this business

# Update AI site with the changes
await update_ai_site(business_id, changes, skip_deploy=True)
# Files collected for combined deploy after Batch 1a + 1b complete