Configuration

Complete reference for environment variables and configuration options

The llms.txt Generator is configured through environment variables. This guide covers all available settings for both backend and frontend components.

Backend Configuration

Backend configuration is managed through a .env file in the backend/ directory.

Required Settings

CORS_ORIGINS · string · default: "http://localhost:3000"

Comma-separated list of allowed CORS origins.

.env

CORS_ORIGINS=http://localhost:3000,https://yourdomain.com

The frontend URL must be included in this list for WebSocket connections to work.

Database Configuration

Required for auto-update functionality and site tracking.

SUPABASE_URL · string · required

Your Supabase project URL.

.env

SUPABASE_URL=https://your-project.supabase.co

Get this from: Supabase Dashboard → Settings → API → Project URL

SUPABASE_KEY · string · required

Your Supabase anonymous/public key.

.env

SUPABASE_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

Get this from: Supabase Dashboard → Settings → API → anon public

Storage Configuration

Required for hosting generated llms.txt files on a public CDN.

R2_ENDPOINT · string · required

Cloudflare R2 endpoint URL.

.env

R2_ENDPOINT=https://abc123.r2.cloudflarestorage.com

Format: https://<account-id>.r2.cloudflarestorage.com

R2_ACCESS_KEY · string · required

R2 access key ID.

.env

R2_ACCESS_KEY=your-access-key-id

Generate from: Cloudflare Dashboard → R2 → Manage R2 API Tokens

R2_SECRET_KEY · string · required

R2 secret access key.

.env

R2_SECRET_KEY=your-secret-access-key

Keep this secret! Never commit to version control.

R2_BUCKET · string · default: "llms-txt"

Name of the R2 bucket for storing llms.txt files.

.env

R2_BUCKET=llms-txt

Create this bucket in the Cloudflare R2 dashboard before using.

R2_PUBLIC_DOMAIN · string · required

Public domain for accessing R2 files.

.env

R2_PUBLIC_DOMAIN=https://pub-abc123.r2.dev

Options:

• Use R2’s public domain: https://pub-<bucket-id>.r2.dev
• Use a custom domain connected to R2

Brightdata Configuration

Optional proxy service for crawling JavaScript-heavy websites.

BRIGHTDATA_API_KEY · string

Your Brightdata customer ID.

.env

BRIGHTDATA_API_KEY=your-customer-id

Get this from: Brightdata Dashboard → Overview → Customer ID

BRIGHTDATA_PASSWORD · string

Zone password for your Brightdata zone.

.env

BRIGHTDATA_PASSWORD=your-zone-password

Get this from: Brightdata Dashboard → Zones → Zone Password

BRIGHTDATA_ZONE · string · default: "scraping_browser1"

Brightdata zone to use for crawling.

.env

BRIGHTDATA_ZONE=scraping_browser1

Options:

• scraping_browser1 - Standard scraping browser
• Custom zones you’ve created

BRIGHTDATA_ENABLED · boolean · default: true

Global toggle for Brightdata usage.

.env

BRIGHTDATA_ENABLED=true

Individual requests can override this with the useBrightdata parameter.

LLM Enhancement Configuration

Optional AI-powered content optimization using OpenRouter.

LLM_ENHANCEMENT_ENABLED · boolean · default: false

Enable LLM-powered content enhancement.

.env

LLM_ENHANCEMENT_ENABLED=false

Must be explicitly enabled. Requires OPENROUTER_API_KEY to be set.

OPENROUTER_API_KEY · string

OpenRouter API key for LLM enhancement.

.env

OPENROUTER_API_KEY=sk-or-v1-...

Get this from: https://openrouter.ai/keys

OPENROUTER_MODEL · string · default: "x-ai/grok-4.1-fast:free"

OpenRouter model to use for enhancement.

.env

OPENROUTER_MODEL=x-ai/grok-4.1-fast:free

Options:

• x-ai/grok-4.1-fast:free - Grok 4.1 Fast (free tier)
• anthropic/claude-3-5-sonnet - Claude 3.5 Sonnet
• openai/gpt-4-turbo - GPT-4 Turbo

See OpenRouter docs for all models.

LLM_TIMEOUT_SECONDS · float

Timeout for LLM API requests in seconds.

.env

LLM_TIMEOUT_SECONDS=30.0

LLM_MAX_RETRIES · integer · default: 3

Maximum retry attempts for failed LLM requests.

.env

LLM_MAX_RETRIES=3

LLM_TEMPERATURE · float

Temperature parameter for LLM generation (0.0-1.0).

.env

LLM_TEMPERATURE=0.3

• Lower values (0.1-0.3): More consistent, deterministic output
• Higher values (0.7-1.0): More creative, varied output

Security Configuration

API_KEY · string

API key for authenticating WebSocket connections.

.env

API_KEY=your-secure-api-key

Generate a secure key:

openssl rand -base64 32

If not set, WebSocket connections will not require authentication. Only use this for development.

CRON_SECRET · string

Secret token for authenticating cron job requests.

.env

CRON_SECRET=your-cron-secret-token

Required for:

• /internal/cron/recrawl - Scheduled recrawl endpoint

Generate a secure token:

openssl rand -base64 32

Frontend Configuration

Frontend configuration is managed through a .env.local file in the frontend/ directory.

NEXT_PUBLIC_WS_URL · string · required

WebSocket URL for the backend API.

.env.local

# Development
NEXT_PUBLIC_WS_URL=ws://localhost:8000/ws/crawl

# Production
NEXT_PUBLIC_WS_URL=wss://your-backend.com/ws/crawl

Use ws:// for local development and wss:// (secure WebSocket) for production.

NEXT_PUBLIC_API_KEY · string

API key for authenticating with the backend (optional).

.env.local

NEXT_PUBLIC_API_KEY=your-api-key

Only use this for testing. Production should use JWT tokens via the /auth/token endpoint.

Configuration File Reference

The backend uses Pydantic Settings for configuration management:

config.py

from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    # CORS
    cors_origins: str = "http://localhost:3000"
    
    # R2 Storage
    r2_endpoint: str | None = None
    r2_access_key: str | None = None
    r2_secret_key: str | None = None
    r2_bucket: str | None = None
    r2_public_domain: str | None = None
    
    # Database
    supabase_url: str | None = None
    supabase_key: str | None = None
    
    # Security
    cron_secret: str | None = None
    api_key: str | None = None
    
    # Brightdata
    brightdata_api_key: str | None = None
    brightdata_enabled: bool = True
    brightdata_zone: str = "scraping_browser1"
    brightdata_password: str | None = None
    
    # LLM Enhancement
    openrouter_api_key: str | None = None
    openrouter_model: str = "x-ai/grok-4.1-fast:free"
    llm_enhancement_enabled: bool = False
    llm_timeout_seconds: float = 30.0
    llm_max_retries: int = 3
    llm_temperature: float = 0.3

    class Config:
        env_file = ".env"

Environment File Examples

Development Environment

backend/.env

# CORS - Allow local frontend
CORS_ORIGINS=http://localhost:3000

# Database (Optional for development)
# SUPABASE_URL=https://xxx.supabase.co
# SUPABASE_KEY=your-key

# Storage (Optional for development)
# R2_ENDPOINT=https://xxx.r2.cloudflarestorage.com
# R2_ACCESS_KEY=your-key
# R2_SECRET_KEY=your-secret
# R2_BUCKET=llms-txt
# R2_PUBLIC_DOMAIN=https://pub-xxx.r2.dev

# Brightdata (Optional)
# BRIGHTDATA_API_KEY=your-customer-id
# BRIGHTDATA_ENABLED=false
# BRIGHTDATA_ZONE=scraping_browser1
# BRIGHTDATA_PASSWORD=your-password

# Security (Optional for development)
# API_KEY=dev-key-only

Production Environment

backend/.env

# CORS - Production domains
CORS_ORIGINS=https://yourdomain.com,https://www.yourdomain.com

# Database - Required
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

# Storage - Required
R2_ENDPOINT=https://abc123.r2.cloudflarestorage.com
R2_ACCESS_KEY=your-access-key
R2_SECRET_KEY=your-secret-key
R2_BUCKET=llms-txt
R2_PUBLIC_DOMAIN=https://cdn.yourdomain.com

# Brightdata - Recommended
BRIGHTDATA_API_KEY=your-customer-id
BRIGHTDATA_ENABLED=true
BRIGHTDATA_ZONE=scraping_browser1
BRIGHTDATA_PASSWORD=your-zone-password

# LLM Enhancement - Optional
OPENROUTER_API_KEY=sk-or-v1-...
OPENROUTER_MODEL=x-ai/grok-4.1-fast:free
LLM_ENHANCEMENT_ENABLED=true
LLM_TIMEOUT_SECONDS=30.0
LLM_MAX_RETRIES=3
LLM_TEMPERATURE=0.3

# Security - Required
API_KEY=<generated-secure-key>
CRON_SECRET=<generated-secure-token>

Validation

Verify your configuration:

1. Test backend health

   Health Check

   curl http://localhost:8000/health
   

   Expected response:

   {"status": "ok"}
   
2. Test database connection

   If Supabase is configured, the backend will log:

   INFO: Supabase client initialized
   
3. Test R2 storage

   Generate a llms.txt file and verify the hosted URL is accessible.

4. Test authentication

   Get Token

   curl -X POST http://localhost:8000/auth/token \
     -H "X-API-Key: your-api-key"
   

   Should return a JWT token.

Troubleshooting

CORS errors in browser console

Symptom: WebSocket connection failed: CORS error

Solution:

• Add your frontend URL to CORS_ORIGINS in backend .env
• Ensure URLs match exactly (including protocol and port)
• Restart the backend after changing .env

Example

CORS_ORIGINS=http://localhost:3000,https://yourdomain.com

No hosted URL generated

Symptom: Generated llms.txt shows but no public URL

Solution:

• Verify all R2 configuration variables are set:
  • R2_ENDPOINT
  • R2_ACCESS_KEY
  • R2_SECRET_KEY
  • R2_BUCKET
  • R2_PUBLIC_DOMAIN
• Verify the bucket exists in Cloudflare R2
• Check bucket permissions allow public reads

Auto-update not working

Symptom: Sites not recrawling on schedule

Solution:

• Verify SUPABASE_URL and SUPABASE_KEY are set
• Check the crawl_sites table exists in Supabase
• Verify AWS Lambda/cron job is configured
• Check CRON_SECRET matches between backend and Lambda

Brightdata crawls failing

Symptom: “Failed to connect to Brightdata” errors

Solution:

• Verify BRIGHTDATA_API_KEY (customer ID) is correct
• Verify BRIGHTDATA_PASSWORD matches your zone password
• Check BRIGHTDATA_ZONE exists in your Brightdata account
• Ensure you have credits/subscription active

LLM enhancement not working

Symptom: No enhancement despite enabling in UI

Solution:

• Set LLM_ENHANCEMENT_ENABLED=true in backend .env
• Verify OPENROUTER_API_KEY is valid
• Check OpenRouter account has credits
• Review backend logs for LLM errors

Best Practices

Use Environment Variables

Never hardcode secrets in source code. Always use environment variables.

Rotate Keys Regularly

Change API keys, tokens, and secrets periodically (every 90 days).

Separate Environments

Use different credentials for development, staging, and production.

Validate on Deploy

Test all endpoints after deployment to catch configuration issues early.

Next Steps

Web Interface

Learn how to use the web UI with your configuration

API Usage

Integrate programmatically using the WebSocket API

Deployment

Deploy to production with proper configuration

Development

Set up a local development environment