Skip to main content

​
Architecture

The generate-all endpoint is a thin orchestrator that coordinates internal services. It returns immediately and runs all tasks in the background.

​
Execution Groups

GroupServicesWaits ForPurpose
1aScrape WebsiteNothingGet pages for GROUP 2 services
1bDiscover CompetitorsNothingFire-and-forget, doesn’t block
1cExa PromptsNothingFire-and-forget, doesn’t block
1dFetch FaviconNothingFire-and-forget, doesn’t block
1eMaterialize ScoreNothingFire-and-forget, doesn’t block
1fSetup CloudFrontNothingFire-and-forget, doesn’t block
2aAI Website, Business PromptsGroup 1aUses scraped pages
2bDiscover ProductsGroup 1aUses scraped pages
3aProduct PromptsGroup 2bUses discovered products
3bGenerate Product LLMsGroup 2bUses discovered products
GROUP 3a and 3b run in parallel after products are discovered. This means product prompts and product llms.txt files are generated simultaneously.
Groups 1b-1f run in parallel with 1a but don’t block Group 2. This means the main onboarding flow (scrape → create AI site → generate prompts) isn’t slowed down by tasks that don’t need scraped pages.

​
Internal Services

Each service is documented on its own page:
ServiceGroupInputOutputPage
Scrape Website1aurlpages[]View →
Discover Competitors1burl, business_namecompetitors[]View →
Exa Prompts1corg_slug, business_name, url10 pre-tested promptsView →
Fetch Favicon1durl, org_slugfavicon URLView →
Materialize Score1eurl, org_slugvisibility scoreView →
Setup CloudFront1furl, org_slugCloudFront distributionView →
Create AI Website2aurl, org_slug, pages[]AI site URLView →
Business Prompts2aurl, org_slug, pages[]40 promptsView →
Discover Products2burl, org_slug, pages[]products[]View →
Product Prompts3aurl, org_slug, products[], pages[]10 prompts/productView →
Generate Product LLMs3burl, org_slug, products[], pages[]llms.txt filesView →

​
Data Flow

scrape_website(url)
    │
    ├── pages[] ──────────────────────────────────────────┐
    │                                                      │
    ▼                                                      ▼
┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│ create_ai_website│  │ business_prompts │  │ discover_products│
│    (pages[])     │  │    (pages[])     │  │    (pages[])     │
└─────────────────┘  └─────────────────┘  └────────┬────────┘
                                                   │
                                         ┌─────────┴─────────┐
                                         ▼                   ▼
                               ┌─────────────────┐  ┌─────────────────┐
                               │ product_prompts │  │generate_product │
                               │   (products[])  │  │   _llms_txt     │
                               │    GROUP 3a     │  │    GROUP 3b     │
                               └─────────────────┘  └─────────────────┘
                                         │                   │
                                         └─────────┬─────────┘
                                                   │
                                              (parallel)

Tasks that DON'T need pages[] (run in parallel with scrape):
┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│   favicon    │  │   scoring    │  │  cloudfront  │
└──────────────┘  └──────────────┘  └──────────────┘
┌──────────────┐  ┌──────────────┐
│ competitors  │  │ exa_prompts  │
└──────────────┘  └──────────────┘

​
Code Location

src/app/apis/onboarding/
├── generate_all/
│   ├── routes.py              # Main orchestrator (GROUP 1a-1f)
│   ├── models.py              # Pydantic models
│   ├── scrape_website.py      # GROUP 1a: Scrape service wrapper
│   ├── exa_prompts.py         # GROUP 1c: Exa prompts service
│   ├── task_orchestrator.py   # GROUP 2 and 3 coordinator
│   └── tasks/                 # Individual task wrappers
│       ├── ai_website.py      # GROUP 2a
│       ├── cloudfront.py      # GROUP 1f
│       ├── discover_products.py # GROUP 2b
│       ├── favicon.py         # GROUP 1d
│       ├── prompts.py         # GROUP 2a + 3a
│       ├── product_llms.py    # GROUP 3b
│       └── scoring.py         # GROUP 1e
└── services/
    └── discover_competitors/  # GROUP 1b: Competitor discovery service

src/app/shared/products/       # Shared services used by both onboarding and cron
├── discover.py                # Product discovery service
└── generate_llms_txt.py       # Product llms.txt generation service

​
Testing Individual Services

Each service can be tested independently via pytest: 
# Test the orchestrator
uv run pytest src/pytests/onboarding/test_generate_all.py -v

# Test shared services used by generate-all
uv run pytest src/pytests/cron/test_generate_prompts.py -v