> ## Documentation Index
> Fetch the complete documentation index at: https://docs.searchcompany.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Architecture for the Generate Product LLMs endpoint

## Purpose

Generates `/llms/{product-slug}.txt` files for products and deploys them to Vercel. These files provide detailed product information optimized for AI consumption.

## Architecture

```mermaid theme={null}
flowchart TD
    Request["POST /api/cron/generate-product-llms-txt"]
    
    subgraph generation [LLMs Generation]
        FetchProducts["Fetch products from DB"]
        ForEach["For each product"]
        FilterPages["Filter pages by source_urls"]
        CallGemini["Call Gemini to generate content"]
    end
    
    subgraph deploy [Deployment]
        CollectFiles["Collect all files"]
        DeployVercel["Deploy to Vercel"]
    end
    
    Request --> FetchProducts
    FetchProducts --> ForEach
    ForEach --> FilterPages
    FilterPages --> CallGemini
    CallGemini --> CollectFiles
    CollectFiles --> DeployVercel
    DeployVercel --> Response["Return counts"]
```

## Pipeline Flow

This endpoint is the third step in a 3-step product pipeline:

```
1. discover-products
   └── Extract products from content
   └── Return entity_ids

2. generate-product-prompts
   └── Takes entity_ids from step 1
   └── Generates 10 prompts per product

3. generate-product-llms-txt (this endpoint)
   └── Takes entity_ids from step 1
   └── Generates /llms/{slug}.txt files
   └── Deploys to Vercel
```

## File Structure

| File         | Path               | Content                   |
| ------------ | ------------------ | ------------------------- |
| Root llms    | `/llms.txt`        | Business overview + links |
| Product llms | `/llms/{slug}.txt` | Product-specific content  |

Example:

```
/llms.txt              → Business overview
/llms/teddy-bear.txt   → Teddy Bear product details
/llms/robot-dog.txt    → Robot Dog product details
```

## Source URL Filtering

The service only uses content from the product's `source_urls`:

```python theme={null}
# Product has source_urls: ["/products/teddy", "/toys"]
# Pages available: ["/", "/about", "/products/teddy", "/toys", "/contact"]
# Filtered pages: ["/products/teddy", "/toys"]
```

This ensures each product llms file contains only relevant content.

## Scalability

This architecture scales to thousands of products:

* Root `llms.txt` stays small (\~3KB) with just links
* Each product gets its own file (\~1-2KB)
* AI crawlers can fetch specific product files on-demand

## Code Location

```
src/app/apis/cron/generate_product_llms_txt/routes.py
src/app/shared/products/generate_llms_txt.py
```