Domain Overview

The Domain API handles connecting customer domains to our CloudFront proxy. This enables AI bots to see optimized content while human visitors see the normal website.

Services Used

Service	Purpose
AWS CloudFront	CDN and edge proxy - routes traffic based on bot detection
AWS Lambda@Edge	Runs at CloudFront edge to detect AI bots and rewrite origins
AWS ACM	SSL certificate management (stores Let’s Encrypt + native certs)
AWS Global Accelerator	Static IPs for apex domain support (A records)
Let’s Encrypt	Initial SSL certificates (bypasses CAA restrictions)
Entri	One-click DNS configuration for customers (no manual record editing)

Architecture

                    ┌─────────────────────────────────────────────┐
                    │              CloudFront Edge                │
                    │         (Lambda@Edge at origin-request)     │
                    └─────────────────────────────────────────────┘
                                        │
                    ┌───────────────────┴───────────────────┐
                    │                                       │
              AI Bot Request                         Human Request
                    │                                       │
                    ▼                                       ▼
        ┌───────────────────┐                   ┌───────────────────┐
        │   AI Site (Vercel)│                   │   Origin Website  │
        │ org-slug.search   │                   │ (customer's host) │
        │ company.dev       │                   │                   │
        └───────────────────┘                   └───────────────────┘

Connection Flow

The domain connection is a two-part process designed for zero downtime:

Part 1: SSL Certificate + Google TXT

User clicks “Start Secure Connection”
We request a Let’s Encrypt certificate via DNS-01 challenge
We also request a Google verification token
User adds BOTH TXT records via Entri (one-click DNS):
- SSL validation TXT record
- Google verification TXT record
Let’s Encrypt validates and issues certificate
Certificate is imported to ACM and attached to CloudFront

Part 2: Connect Domain + Verify Google

SSL is ready on CloudFront
User clicks “Point Domain”
User switches www CNAME from origin to CloudFront via Entri
Traffic now flows through our proxy
Background tasks trigger:
- ACM-native certificate upgrade (auto-renewal)
- Google verification polling (10s × 12 = 2 min)
- Once Google verified → IndexNow + GSC sitemap submission
- If Google fails → IndexNow only (GSC skipped)

Status Flow

PENDING_VALIDATION  →  SSL_VALIDATING  →  SSL_VALIDATED  →  DEPLOYED
      │                      │                  │              │
      │                      │                  │              │
  CloudFront             TXT record          Certificate    www CNAME
   created              added, waiting       attached to    points to
                        for Let's Encrypt    CloudFront     CloudFront
                                                               │
                                                               ▼
                                                         DISCONNECTED
                                                               │
                                                         (can reconnect
                                                          instantly)

Endpoints

Endpoint	Purpose
Get Proxy	Get current proxy status and DNS records
Start Certificate	Begin Let’s Encrypt certificate request
Complete Certificate	Finish certificate after TXT record added
Start Google Verification	Get Google TXT verification token (Step 1)
Complete Google Verification	Verify domain + add to Search Console (now backend-driven in Step 2)
Resubmit Sitemap	Resubmit sitemap after new boosted pages (cron)
Mark Step Complete	Update status after Entri success + trigger Google verification polling
Verify DNS	Live DNS lookup verification
Disconnect Proxy	Get DNS records to restore original config
Mark Disconnect Complete	Update status after disconnect completes
Update Lambda@Edge	Deploy new bot patterns to all distributions

Setup CloudFront Proxy is now part of the Onboarding flow - it runs automatically during generate-all.

Why Let’s Encrypt + ACM?

We use a hybrid approach for SSL certificates: Initial: Let’s Encrypt

Works with ANY DNS provider
Bypasses CAA restrictions (Vercel/Netlify block ACM)
User adds one TXT record, done

After Connection: ACM-Native

Once www points to CloudFront, ACM can validate
Background upgrade happens automatically
ACM-native certs auto-renew forever
User never has to touch DNS again

Lambda@Edge Bot Detection

The Lambda@Edge function at origin-request detects AI bots by User-Agent:

// Detected AI bots route to AI site
const AI_BOTS = [
  'GPTBot', 'ChatGPT', 'Claude', 'Anthropic',
  'Perplexity', 'Google-Extended', 'Bingbot',
  'CCBot', 'cohere-ai', 'Amazonbot'
];

// Human traffic routes to origin website

Traffic Type	Destination	Host Header
AI Bot	`org-slug.searchcompany.dev`	AI site host
Human	Customer’s origin	Original domain

IndexNow Key File Routing

The Lambda also routes /search-company.txt to the AI site regardless of User-Agent. This ensures IndexNow can verify domain ownership when we submit URLs. Without this, IndexNow’s verification request would go to the human website (which doesn’t have the key file) and fail.

Bot Patterns - Single Source of Truth

Bot patterns are defined in one place: src/app/shared/bot_identifiers.py This file is used by:

AI Recommendation Store - Identifying which AI platform visited
Lambda@Edge - Generated via generate_lambda_code.py

Updating Bot Patterns

When new AI bots emerge:

# 1. Edit the shared source
vim src/app/shared/bot_identifiers.py

# 2. Regenerate Lambda code
uv run python src/app/domain/generate_lambda_code.py

# 3. Deploy to all distributions
curl -X POST https://api.searchcompany.ai/api/domain/update-lambda-edge \
  -H "Authorization: Bearer $TOKEN"

All customer distributions update automatically (~15 min for 1,000 customers). Zero downtime - old version runs until new one propagates.

Database Schema

The ai_sites table stores all proxy configuration:

Column	Purpose
`entity_id`	Links to business entity
`cloudfront_distribution_id`	CloudFront distribution ID
`cloudfront_domain`	e.g., `d123abc.cloudfront.net`
`custom_domain`	e.g., `www.example.com`
`origin_cname`	Where www originally pointed
`original_www_cname`	Preserved for disconnect
`certificate_arn`	ACM certificate ARN
`certificate_type`	`LETS_ENCRYPT` or `ACM_NATIVE`
`proxy_status`	Current status
`le_*`	Let’s Encrypt temp data
`google_verification_token`	TXT record value from Google Site Verification API
`google_verification_status`	`PENDING`, `VERIFIED`, or `FAILED`
`google_sitemap_submitted_at`	Last sitemap submission timestamp

Disconnect and Relink Flow

Users can disconnect their domain and reconnect later:

Disconnect Flow

User clicks “Disconnect Domain”
Frontend calls /disconnect-proxy → gets restore DNS records
Entri restores www CNAME to original
Frontend calls /mark-disconnect-complete → status becomes DISCONNECTED
CloudFront + ACM stay intact (for fast relink)

Relink Flow (Fast)

User clicks “Reconnect”
No CloudFront setup needed - distribution already exists
Start certificate (if expired) or skip
2-step DNS flow: SSL CNAME → www CNAME
Status becomes DEPLOYED

This is much faster than initial setup because CloudFront and ACM are preserved.

Testing

The domain connection can be tested end-to-end:

# Check current status
curl https://api.searchcompany.ai/api/domain/get-proxy/org_xxx \
  -H "Authorization: Bearer $TOKEN"

# Start certificate (Part 1)
curl -X POST https://api.searchcompany.ai/api/domain/start-certificate/org_xxx \
  -H "Authorization: Bearer $TOKEN"

# After adding TXT record, complete certificate
curl -X POST https://api.searchcompany.ai/api/domain/complete-certificate/org_xxx \
  -H "Authorization: Bearer $TOKEN"

# After switching www CNAME (Part 2)
curl -X POST https://api.searchcompany.ai/api/domain/mark-step-complete/org_xxx \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"step": 2}'

# Disconnect domain
curl -X POST https://api.searchcompany.ai/api/domain/disconnect-proxy/org_xxx \
  -H "Authorization: Bearer $TOKEN"

# After Entri restores DNS
curl -X POST https://api.searchcompany.ai/api/domain/mark-disconnect-complete/org_xxx \
  -H "Authorization: Bearer $TOKEN"

Getting Started

Website

Onboarding

Cron

Your Current Setup

What We're Doing

Explore

Settings - Toggle

Settings - Business

Settings - Team

Settings - Billing

Settings - Domain

Webhooks

Health

Manual Trigger

Services Used

Architecture

Connection Flow

Part 1: SSL Certificate + Google TXT

Part 2: Connect Domain + Verify Google

Status Flow

Endpoints

Why Let’s Encrypt + ACM?

Lambda@Edge Bot Detection

IndexNow Key File Routing

Bot Patterns - Single Source of Truth

Updating Bot Patterns

Database Schema

Disconnect and Relink Flow

Disconnect Flow

Relink Flow (Fast)

Testing

Getting Started

Website

Onboarding

Cron

Your Current Setup

What We're Doing

Explore

Settings - Toggle

Settings - Business

Settings - Team

Settings - Billing

Settings - Domain

Webhooks

Health

Manual Trigger

​Services Used

​Architecture

​Connection Flow

​Part 1: SSL Certificate + Google TXT

​Part 2: Connect Domain + Verify Google

​Status Flow

​Endpoints

​Why Let’s Encrypt + ACM?

​Lambda@Edge Bot Detection

​IndexNow Key File Routing

​Bot Patterns - Single Source of Truth

​Updating Bot Patterns

​Database Schema

​Disconnect and Relink Flow

​Disconnect Flow

​Relink Flow (Fast)

​Testing

Services Used

Architecture

Connection Flow

Part 1: SSL Certificate + Google TXT

Part 2: Connect Domain + Verify Google

Status Flow

Endpoints

Why Let’s Encrypt + ACM?

Lambda@Edge Bot Detection

IndexNow Key File Routing

Bot Patterns - Single Source of Truth

Updating Bot Patterns

Database Schema

Disconnect and Relink Flow

Disconnect Flow

Relink Flow (Fast)

Testing