> ## Documentation Index
> Fetch the complete documentation index at: https://docs.searchcompany.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Search Engine Submission

> Background services that submit URLs to IndexNow and Google Search Console

# Internal Services: IndexNow + GSC Submission

After Step 2 of domain connection completes, a background orchestrator handles Google verification and search engine submission:

1. **Google Verification** - Polls TXT record verification (10s × 12 = 2 min)
2. **IndexNow** - Notifies Bing, Yandex, and other IndexNow-compatible engines
3. **Google Search Console** - Submits sitemap (only if domain is verified)

## Purpose

These tasks ensure search engines are immediately notified about the AI site after the proxy is connected. This is one of only TWO places where search engine submission happens:

1. **Here** - One-time after domain connection (Step 2)
2. **BATCH 3 in daily cron** - After new content is deployed

## Why Poll Google Verification in Step 2?

The Google TXT record is added during Step 1 (alongside the SSL TXT record), but we verify it in Step 2 because:

1. **DNS Propagation**: TXT records need time to propagate globally
2. **More Time**: By Step 2, the record has had time to propagate
3. **Resilience**: Polling handles slow DNS gracefully (up to 2 minutes)
4. **Non-blocking**: User doesn't wait - it runs in background

## Function Signatures

```python theme={null}
async def _background_google_verify_and_submit(
    ai_site_id: str,
    entity_id: str,
    org_id: str,
    source_url: str,
    custom_domain: str,
    google_verification_token: str
)

async def _background_indexnow_submission(
    entity_id: str,
    org_id: str,
    source_url: str
)

async def _background_gsc_submission(
    ai_site_id: str,
    org_id: str,
    source_url: str
)
```

## Parameters

### Google Verify + Submit Orchestrator

| Parameter                   | Type | Description                                                                |
| --------------------------- | ---- | -------------------------------------------------------------------------- |
| `ai_site_id`                | str  | AI site database ID                                                        |
| `entity_id`                 | str  | Database entity ID                                                         |
| `org_id`                    | str  | Clerk organization ID                                                      |
| `source_url`                | str  | Customer's source URL (e.g., "[https://example.com](https://example.com)") |
| `custom_domain`             | str  | The domain to verify (e.g., "[www.example.com](http://www.example.com)")   |
| `google_verification_token` | str  | Token from Step 1's start-google-verification                              |

### IndexNow

| Parameter    | Type | Description                                                                |
| ------------ | ---- | -------------------------------------------------------------------------- |
| `entity_id`  | str  | Database entity ID                                                         |
| `org_id`     | str  | Clerk organization ID                                                      |
| `source_url` | str  | Customer's source URL (e.g., "[https://example.com](https://example.com)") |

### GSC

| Parameter    | Type | Description           |
| ------------ | ---- | --------------------- |
| `ai_site_id` | str  | AI site database ID   |
| `org_id`     | str  | Clerk organization ID |
| `source_url` | str  | Customer's source URL |

## URLs Submitted (IndexNow)

| Category       | Source                                        |
| -------------- | --------------------------------------------- |
| Sitemap URLs   | Fetched from deployed AI site's `sitemap.xml` |
| `/sitemap.xml` | Added explicitly (not in sitemap itself)      |
| Boosted pages  | All paths from `boosted_pages.deployment_url` |

The function fetches `sitemap.xml` from the deployed AI site, parses all `<loc>` URLs, replaces the AI site domain with the customer's source domain, and submits to IndexNow.

## Execution Flow

```mermaid theme={null}
flowchart TD
    A[Step 2 Complete] --> B[Start Background Orchestrator]
    B --> C[Poll Google Verification]
    C --> D{Attempt 1-12<br/>10s intervals}
    D -->|Verified| E[Update DB: VERIFIED]
    D -->|Not Verified| F{More attempts?}
    F -->|Yes| D
    F -->|No after 2 min| G[Update DB: FAILED]
    E --> H[Add to Search Console]
    H --> I[IndexNow Submission]
    G --> I
    I --> J[Build URL List]
    J --> K[Submit to IndexNow API]
    E --> L[GSC Sitemap Submission]
    L --> M[Submit sitemap.xml]
```

## Polling Details

| Setting       | Value          |
| ------------- | -------------- |
| Poll interval | 10 seconds     |
| Max attempts  | 12             |
| Total timeout | 2 minutes      |
| On success    | IndexNow + GSC |
| On failure    | IndexNow only  |

## Dependencies

* `src/app/shared/google_search_console.py` - `verify_domain()`, `add_to_search_console()`, `submit_sitemap()`
* `src/app/shared/indexnow.py` - `submit_urls_to_indexnow()`

## Code Location

```
src/app/apis/domain/mark_step_complete/routes.py
```