Overview

​

Purpose

1. Hashing Service - Fetch raw HTML directly (free) and compare hashes
2. Batch Scrape - Only scrape pages that actually changed (paid)

​

Architecture

​

Detection Logic

1. Get stored data - Retrieve site_map (URL list) and page_hashes from ai_sites
2. Map current URLs - Use custom mapper to get current URL list [FREE]
3. Compare URL lists - Find new URLs, removed URLs, existing URLs
4. Fetch + Hash existing pages - Use hashing service to get raw HTML and hash [FREE]
5. Compare hashes - Find which existing pages changed
6. Batch scrape - Only scrape NEW + CHANGED pages with Firecrawl [PAID]

​

Cost Optimization

• Old approach: 100 × $0.01 =$1.00
• New approach: 3 × $0.01 =$0.03

​

Response Format

{
  "status": "success",
  "new_pages": [...],        // Have markdown (batch scraped)
  "changed_pages": [...],    // Have markdown (batch scraped)
  "removed_urls": [...],
  "unchanged_pages": [...],  // NO markdown (not scraped)
  "updated_site_map": [...],
  "updated_hashes": {...},
  "business_info": {...}
}

​

Change Categories

​

Code Location

src/app/apis/cron/detect_changes/
├── routes.py           # HTTP endpoint
└── mini_orchestrator.py # Detection logic

src/app/shared/hashing/
├── __init__.py         # Exports
└── service.py          # fetch_and_hash, fetch_and_hash_batch

src/app/shared/mapping/
├── __init__.py         # Exports
└── mapper.py           # map_website

​

Used By

• Daily cron job for site freshness
• Triggers update-ai-site when changes found
• Triggers discover-products-from-changes for new pages