> ## Documentation Index
> Fetch the complete documentation index at: https://docs.searchcompany.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Search Console Overview

> Google Search Console integration for AI visibility with Google AI Overviews

Google Search Console (GSC) integration allows us to verify customer domains with Google and submit sitemaps for better indexing. This is critical for visibility in **Google AI Overviews**, as Google does not support IndexNow.

## Why Google Search Console?

| Search Engine | Indexing Method    | AI Product   |
| ------------- | ------------------ | ------------ |
| **Google**    | Search Console API | AI Overviews |
| **Bing**      | IndexNow           | Copilot      |
| **Yandex**    | IndexNow           | N/A          |

Google is the only major search engine that doesn't support IndexNow. To get our AI-optimized pages indexed by Google (and visible in AI Overviews), we must use the Search Console API.

## Architecture

We use a **master account** approach:

```
┌─────────────────────────────────────────────────────────────┐
│                    admin_1@searchcompany.co                 │
│                    (Master GSC Account)                     │
│                                                             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │ customer1   │  │ customer2   │  │ customer3   │  ...    │
│  │ .com        │  │ .io         │  │ .co         │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
│                                                             │
│              Up to 1,000 sites per account                  │
└─────────────────────────────────────────────────────────────┘
```

* **One Google account** owns all verified customer domains
* Customers don't need their own Google accounts
* We use OAuth2 with a **refresh token** for API access
* The refresh token never expires (as long as used every 6 months)

## Limits

<Warning>
  Each Google account can manage a maximum of **1,000 sites** in Search Console.
  When approaching this limit, you must add a new admin account (see below).
</Warning>

## Current Setup

| Environment Variable               | Description                                  |
| ---------------------------------- | -------------------------------------------- |
| `GOOGLE_CLIENT_ID_GSC`             | OAuth client ID from Google Cloud Console    |
| `GOOGLE_CLIENT_SECRET_GSC`         | OAuth client secret                          |
| `GOOGLE_REFRESH_TOKEN_GSC_ADMIN_1` | Refresh token for `admin_1@searchcompany.co` |

## Adding a New Admin Account

When you reach \~900 sites on `admin_1`, it's time to add `admin_2@searchcompany.co`.

### Step 1: Create the Google Account

1. Create a new Google Workspace account: `admin_2@searchcompany.co`
2. Ensure it has access to Google Search Console

### Step 2: Get OAuth Credentials

The same OAuth client (Client ID + Secret) can be used for multiple accounts. You only need a new **refresh token**.

1. Go to [OAuth Playground](https://developers.google.com/oauthplayground)
2. Click the **gear icon** (⚙️) → Check "Use your own OAuth credentials"
3. Enter the existing `GOOGLE_CLIENT_ID_GSC` and `GOOGLE_CLIENT_SECRET_GSC`
4. Select scopes:
   * `https://www.googleapis.com/auth/siteverification`
   * `https://www.googleapis.com/auth/webmasters`
5. Click "Authorize APIs"
6. **Sign in as `admin_2@searchcompany.co`** (not admin\_1)
7. Grant permissions
8. Click "Exchange authorization code for tokens"
9. Copy the **Refresh Token**

### Step 3: Add Environment Variable

Add the new refresh token to your `.env`:

```bash theme={null}
# Google Search Console OAuth
GOOGLE_CLIENT_ID_GSC=your-client-id.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET_GSC=your-client-secret
GOOGLE_REFRESH_TOKEN_GSC_ADMIN_1=1//token-for-admin-1
GOOGLE_REFRESH_TOKEN_GSC_ADMIN_2=1//token-for-admin-2  # NEW
```

### Step 4: Update the Code

Modify `Backend/src/app/shared/google_search_console/client.py` to support multiple accounts:

```python theme={null}
def _get_credentials(account: str = "ADMIN_1") -> tuple[str, str, str]:
    """Get OAuth credentials from environment."""
    client_id = os.getenv("GOOGLE_CLIENT_ID_GSC")
    client_secret = os.getenv("GOOGLE_CLIENT_SECRET_GSC")
    refresh_token = os.getenv(f"GOOGLE_REFRESH_TOKEN_GSC_{account}")

    if not all([client_id, client_secret, refresh_token]):
        raise ValueError(f"Missing Google OAuth credentials for {account}")

    return client_id, client_secret, refresh_token
```

### Step 5: Implement Account Selection Logic

Add logic to select which account to use based on current site counts:

```python theme={null}
async def get_account_for_new_site() -> str:
    """Determine which admin account to use for a new site."""
    # Check site counts for each account
    admin_1_count = await count_sites_for_account("ADMIN_1")

    if admin_1_count < 950:
        return "ADMIN_1"

    admin_2_count = await count_sites_for_account("ADMIN_2")
    if admin_2_count < 950:
        return "ADMIN_2"

    # Add more accounts as needed
    raise Exception("All GSC accounts at capacity")
```

### Step 6: Store Account Assignment

Add a column to `ai_sites` table to track which account owns each site:

```sql theme={null}
ALTER TABLE ai_sites ADD COLUMN google_admin_account TEXT DEFAULT 'ADMIN_1';
```

## Token Refresh Behavior

The refresh token stays valid as long as:

1. **Used at least once every 6 months** - Our daily cron (sitemap resubmission) handles this automatically
2. **User doesn't revoke access** - Only if someone logs into the Google account and revokes app access
3. **Under 50 refresh tokens per account** - We only have 1 per account, so no issue

<Note>
  If a refresh token is revoked or expires, you'll need to repeat the OAuth
  Playground flow to get a new one.
</Note>

## Endpoints

| Endpoint                                                                                    | Purpose                                                |
| ------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| [Start Google Verification](/api-reference/endpoint/domain/start-google-verification)       | Get TXT verification token from Google                 |
| [Complete Google Verification](/api-reference/endpoint/domain/complete-google-verification) | Verify domain + add to Search Console + submit sitemap |
| [Resubmit Sitemap](/api-reference/endpoint/domain/resubmit-sitemap)                         | Resubmit sitemap after new boosted pages (cron)        |

## Flow During Domain Connection

Google verification runs in a **two-step process**:

### Step 1: Get TXT Records (Frontend)

```
User clicks "Start Secure Connection"
           │
           ├──────────────────────────────────────┐
           │                                      │
           ▼                                      ▼
   /start-certificate                /start-google-verification
   (Let's Encrypt TXT)               (Google TXT)
           │                                      │
           └──────────────────────────────────────┘
                              │
                              ▼
                     Entri shows BOTH
                     TXT records to add
                              │
                              ▼
                     User adds via Entri
                              │
                              ▼
                    /complete-certificate
                    (Issue + attach cert)
```

### Step 2: Verify Google + Submit (Backend)

```
User clicks "Point Domain" (switches CNAME)
                              │
                              ▼
                    /mark-step-complete
                    (status = DEPLOYED)
                              │
                              ▼
              Background: Poll Google Verification
              (10 seconds × 12 = 2 minutes max)
                              │
           ┌──────────────────┴──────────────────┐
           │                                      │
       Verified                              Not Verified
           │                                      │
           ▼                                      ▼
   Add to Search Console                   Skip GSC
   Submit sitemap                          (IndexNow still runs)
```

<Note>
  **Why poll in Step 2?** The Google TXT record is added during Step 1, but DNS
  propagation takes time. By polling in Step 2 (after the user completes the
  CNAME switch), we give DNS more time to propagate. The backend polls every 10
  seconds for up to 2 minutes.
</Note>

<Note>
  Google verification failure is **non-blocking**. If it fails, the domain is
  still connected and IndexNow submission still runs. Only GSC sitemap
  submission is skipped.
</Note>

## Database Columns

The `ai_sites` table stores Google verification state:

| Column                        | Type        | Purpose                                            |
| ----------------------------- | ----------- | -------------------------------------------------- |
| `google_verification_token`   | TEXT        | TXT record value from Google Site Verification API |
| `google_verification_status`  | TEXT        | `PENDING`, `VERIFIED`, or `FAILED`                 |
| `google_sitemap_submitted_at` | TIMESTAMPTZ | Last sitemap submission timestamp                  |
| `google_admin_account`        | TEXT        | Which admin account owns this site (future)        |

## Troubleshooting

### "Missing Google OAuth credentials"

Check that all three environment variables are set:

* `GOOGLE_CLIENT_ID_GSC`
* `GOOGLE_CLIENT_SECRET_GSC`
* `GOOGLE_REFRESH_TOKEN_GSC_ADMIN_1`

### "Failed to get access token"

The refresh token may be invalid. Re-run the OAuth Playground flow to get a new one.

### "Domain verification failed"

1. Check that the TXT record was added correctly
2. DNS propagation can take up to 48 hours (usually 5-30 minutes)
3. Verify the TXT record with: `dig TXT example.com`

### "Failed to add site to Search Console"

The domain may already be verified by another account. Check Search Console manually.