Search Console Overview

Google Search Console (GSC) integration allows us to verify customer domains with Google and submit sitemaps for better indexing. This is critical for visibility in Google AI Overviews, as Google does not support IndexNow.

Why Google Search Console?

Search Engine	Indexing Method	AI Product
Google	Search Console API	AI Overviews
Bing	IndexNow	Copilot
Yandex	IndexNow	N/A

Google is the only major search engine that doesn’t support IndexNow. To get our AI-optimized pages indexed by Google (and visible in AI Overviews), we must use the Search Console API.

Architecture

We use a master account approach:

┌─────────────────────────────────────────────────────────────┐
│                    [email protected]                 │
│                    (Master GSC Account)                     │
│                                                             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │ customer1   │  │ customer2   │  │ customer3   │  ...    │
│  │ .com        │  │ .io         │  │ .co         │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
│                                                             │
│              Up to 1,000 sites per account                  │
└─────────────────────────────────────────────────────────────┘

One Google account owns all verified customer domains
Customers don’t need their own Google accounts
We use OAuth2 with a refresh token for API access
The refresh token never expires (as long as used every 6 months)

Limits

Each Google account can manage a maximum of 1,000 sites in Search Console. When approaching this limit, you must add a new admin account (see below).

Current Setup

Environment Variable	Description
`GOOGLE_CLIENT_ID_GSC`	OAuth client ID from Google Cloud Console
`GOOGLE_CLIENT_SECRET_GSC`	OAuth client secret
`GOOGLE_REFRESH_TOKEN_GSC_ADMIN_1`	Refresh token for `[email protected]`

Adding a New Admin Account

When you reach ~900 sites on admin_1, it’s time to add [email protected].

Step 1: Create the Google Account

Create a new Google Workspace account: [email protected]
Ensure it has access to Google Search Console

Step 2: Get OAuth Credentials

The same OAuth client (Client ID + Secret) can be used for multiple accounts. You only need a new refresh token.

Go to OAuth Playground
Click the gear icon (⚙️) → Check “Use your own OAuth credentials”
Enter the existing GOOGLE_CLIENT_ID_GSC and GOOGLE_CLIENT_SECRET_GSC
Select scopes:
- https://www.googleapis.com/auth/siteverification
- https://www.googleapis.com/auth/webmasters
Click “Authorize APIs”
Sign in as [email protected] (not admin_1)
Grant permissions
Click “Exchange authorization code for tokens”
Copy the Refresh Token

Step 3: Add Environment Variable

Add the new refresh token to your .env:

# Google Search Console OAuth
GOOGLE_CLIENT_ID_GSC=your-client-id.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET_GSC=your-client-secret
GOOGLE_REFRESH_TOKEN_GSC_ADMIN_1=1//token-for-admin-1
GOOGLE_REFRESH_TOKEN_GSC_ADMIN_2=1//token-for-admin-2  # NEW

Step 4: Update the Code

Modify Backend/src/app/shared/google_search_console/client.py to support multiple accounts:

def _get_credentials(account: str = "ADMIN_1") -> tuple[str, str, str]:
    """Get OAuth credentials from environment."""
    client_id = os.getenv("GOOGLE_CLIENT_ID_GSC")
    client_secret = os.getenv("GOOGLE_CLIENT_SECRET_GSC")
    refresh_token = os.getenv(f"GOOGLE_REFRESH_TOKEN_GSC_{account}")

    if not all([client_id, client_secret, refresh_token]):
        raise ValueError(f"Missing Google OAuth credentials for {account}")

    return client_id, client_secret, refresh_token

Step 5: Implement Account Selection Logic

Add logic to select which account to use based on current site counts:

async def get_account_for_new_site() -> str:
    """Determine which admin account to use for a new site."""
    # Check site counts for each account
    admin_1_count = await count_sites_for_account("ADMIN_1")

    if admin_1_count < 950:
        return "ADMIN_1"

    admin_2_count = await count_sites_for_account("ADMIN_2")
    if admin_2_count < 950:
        return "ADMIN_2"

    # Add more accounts as needed
    raise Exception("All GSC accounts at capacity")

Step 6: Store Account Assignment

Add a column to ai_sites table to track which account owns each site:

ALTER TABLE ai_sites ADD COLUMN google_admin_account TEXT DEFAULT 'ADMIN_1';

Token Refresh Behavior

The refresh token stays valid as long as:

Used at least once every 6 months - Our daily cron (sitemap resubmission) handles this automatically
User doesn’t revoke access - Only if someone logs into the Google account and revokes app access
Under 50 refresh tokens per account - We only have 1 per account, so no issue

If a refresh token is revoked or expires, you’ll need to repeat the OAuth Playground flow to get a new one.

Endpoints

Endpoint	Purpose
Start Google Verification	Get TXT verification token from Google
Complete Google Verification	Verify domain + add to Search Console + submit sitemap
Resubmit Sitemap	Resubmit sitemap after new boosted pages (cron)

Flow During Domain Connection

Google verification runs in a two-step process:

Step 1: Get TXT Records (Frontend)

User clicks "Start Secure Connection"
           │
           ├──────────────────────────────────────┐
           │                                      │
           ▼                                      ▼
   /start-certificate                /start-google-verification
   (Let's Encrypt TXT)               (Google TXT)
           │                                      │
           └──────────────────────────────────────┘
                              │
                              ▼
                     Entri shows BOTH
                     TXT records to add
                              │
                              ▼
                     User adds via Entri
                              │
                              ▼
                    /complete-certificate
                    (Issue + attach cert)

Step 2: Verify Google + Submit (Backend)

User clicks "Point Domain" (switches CNAME)
                              │
                              ▼
                    /mark-step-complete
                    (status = DEPLOYED)
                              │
                              ▼
              Background: Poll Google Verification
              (10 seconds × 12 = 2 minutes max)
                              │
           ┌──────────────────┴──────────────────┐
           │                                      │
       Verified                              Not Verified
           │                                      │
           ▼                                      ▼
   Add to Search Console                   Skip GSC
   Submit sitemap                          (IndexNow still runs)

Why poll in Step 2? The Google TXT record is added during Step 1, but DNS propagation takes time. By polling in Step 2 (after the user completes the CNAME switch), we give DNS more time to propagate. The backend polls every 10 seconds for up to 2 minutes.

Google verification failure is non-blocking. If it fails, the domain is still connected and IndexNow submission still runs. Only GSC sitemap submission is skipped.

Database Columns

The ai_sites table stores Google verification state:

Column	Type	Purpose
`google_verification_token`	TEXT	TXT record value from Google Site Verification API
`google_verification_status`	TEXT	`PENDING`, `VERIFIED`, or `FAILED`
`google_sitemap_submitted_at`	TIMESTAMPTZ	Last sitemap submission timestamp
`google_admin_account`	TEXT	Which admin account owns this site (future)

Troubleshooting

”Missing Google OAuth credentials”

Check that all three environment variables are set:

GOOGLE_CLIENT_ID_GSC
GOOGLE_CLIENT_SECRET_GSC
GOOGLE_REFRESH_TOKEN_GSC_ADMIN_1

”Failed to get access token”

The refresh token may be invalid. Re-run the OAuth Playground flow to get a new one.

”Domain verification failed”

Check that the TXT record was added correctly
DNS propagation can take up to 48 hours (usually 5-30 minutes)
Verify the TXT record with: dig TXT example.com

”Failed to add site to Search Console”

The domain may already be verified by another account. Check Search Console manually.

Getting Started

Website

Onboarding

Cron

Your Current Setup

What We're Doing

Explore

Settings - Toggle

Settings - Business

Settings - Team

Settings - Billing

Settings - Domain

Webhooks

Health

Manual Trigger

Why Google Search Console?

Architecture

Limits

Current Setup

Adding a New Admin Account

Step 1: Create the Google Account

Step 2: Get OAuth Credentials

Step 3: Add Environment Variable

Step 4: Update the Code

Step 5: Implement Account Selection Logic

Step 6: Store Account Assignment

Token Refresh Behavior

Endpoints

Flow During Domain Connection

Step 1: Get TXT Records (Frontend)

Step 2: Verify Google + Submit (Backend)

Database Columns

Troubleshooting

”Missing Google OAuth credentials”

”Failed to get access token”

”Domain verification failed”

”Failed to add site to Search Console”

Getting Started

Website

Onboarding

Cron

Your Current Setup

What We're Doing

Explore

Settings - Toggle

Settings - Business

Settings - Team

Settings - Billing

Settings - Domain

Webhooks

Health

Manual Trigger

​Why Google Search Console?

​Architecture

​Limits

​Current Setup

​Adding a New Admin Account

​Step 1: Create the Google Account

​Step 2: Get OAuth Credentials

​Step 3: Add Environment Variable

​Step 4: Update the Code

​Step 5: Implement Account Selection Logic

​Step 6: Store Account Assignment

​Token Refresh Behavior

​Endpoints

​Flow During Domain Connection

​Step 1: Get TXT Records (Frontend)

​Step 2: Verify Google + Submit (Backend)

​Database Columns

​Troubleshooting

​”Missing Google OAuth credentials”

​”Failed to get access token”

​”Domain verification failed”

​”Failed to add site to Search Console”

Why Google Search Console?

Architecture

Limits

Current Setup

Adding a New Admin Account

Step 1: Create the Google Account

Step 2: Get OAuth Credentials

Step 3: Add Environment Variable

Step 4: Update the Code

Step 5: Implement Account Selection Logic

Step 6: Store Account Assignment

Token Refresh Behavior

Endpoints

Flow During Domain Connection

Step 1: Get TXT Records (Frontend)

Step 2: Verify Google + Submit (Backend)

Database Columns

Troubleshooting

”Missing Google OAuth credentials”

”Failed to get access token”

”Domain verification failed”

”Failed to add site to Search Console”