> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/Anwitht21/llmstxt/llms.txt
> Use this file to discover all available pages before exploring further.

# Quick Start

> Get your first llms.txt file generated in 5 minutes with local development setup

# Quick Start Guide

Get the llms.txt Generator running locally in under 5 minutes. This guide will walk you through setting up both the backend and frontend for development.

## Prerequisites

Before you begin, ensure you have the following installed:

<CardGroup cols={3}>
  <Card title="Python 3.11+" icon="python">
    Required for the FastAPI backend
  </Card>

  <Card title="Node.js 20+" icon="node-js">
    Required for the Next.js frontend
  </Card>

  <Card title="Git" icon="git">
    For cloning the repository
  </Card>
</CardGroup>

<Note>
  Docker is optional but recommended for simplified deployment. See the [Docker Setup](#docker-setup-optional) section below.
</Note>

## Installation

<Steps>
  <Step title="Clone the Repository">
    Clone the project to your local machine:

    ```bash theme={null}
    git clone <your-repo-url>
    cd llmstxt
    ```
  </Step>

  <Step title="Backend Setup">
    Set up the Python environment and install dependencies:

    ```bash theme={null}
    cd backend
    python -m venv venv
    source venv/bin/activate  # On Windows: venv\Scripts\activate
    pip install -r requirements.txt
    ```

    The `requirements.txt` includes:

    <CodeGroup>
      ```txt requirements.txt theme={null}
      fastapi
      uvicorn[standard]
      httpx
      beautifulsoup4
      pydantic
      pydantic-settings
      python-dotenv
      boto3
      pytest
      pytest-asyncio
      supabase==2.10.0
      playwright
      PyJWT[crypto]
      ```
    </CodeGroup>

    Install Playwright browsers:

    ```bash theme={null}
    playwright install chromium
    ```
  </Step>

  <Step title="Backend Environment Configuration">
    Create your backend environment file:

    ```bash theme={null}
    cp .env.example .env
    ```

    Edit `.env` with your configuration:

    <CodeGroup>
      ```bash .env theme={null}
      # CORS Configuration
      CORS_ORIGINS=http://localhost:3000

      # Cloudflare R2 Storage (Required)
      R2_ENDPOINT=https://your-account-id.r2.cloudflarestorage.com
      R2_ACCESS_KEY=your-access-key
      R2_SECRET_KEY=your-secret-key
      R2_BUCKET=llms-txt
      R2_PUBLIC_DOMAIN=https://your-public-domain.com

      # Supabase Database (Required)
      SUPABASE_URL=https://your-project.supabase.co
      SUPABASE_KEY=your-anon-key

      # Cron Secret for scheduled updates
      CRON_SECRET=your-secret-token

      # Brightdata Proxy (Optional - for JS-heavy sites)
      BRIGHTDATA_API_KEY=your-customer-id-here
      BRIGHTDATA_ENABLED=true
      BRIGHTDATA_ZONE=scraping_browser1
      BRIGHTDATA_PASSWORD=your-zone-password-here
      ```
    </CodeGroup>

    <Warning>
      At minimum, you need to configure R2 storage and Supabase. The Brightdata proxy is optional and only needed for JavaScript-heavy websites.
    </Warning>
  </Step>

  <Step title="Frontend Setup">
    In a new terminal, navigate to the frontend directory:

    ```bash theme={null}
    cd frontend
    npm install
    ```

    Create the frontend environment file:

    ```bash theme={null}
    cp .env.example .env.local
    ```

    Edit `.env.local`:

    <CodeGroup>
      ```bash .env.local theme={null}
      NEXT_PUBLIC_WS_URL=ws://localhost:8000/ws/crawl
      ```
    </CodeGroup>
  </Step>

  <Step title="Start the Servers">
    Start both servers in separate terminals:

    <CodeGroup>
      ```bash Backend (Terminal 1) theme={null}
      cd backend
      source venv/bin/activate
      uvicorn main:app --reload --port 8000
      ```

      ```bash Frontend (Terminal 2) theme={null}
      cd frontend
      npm run dev
      ```
    </CodeGroup>

    You should see:

    * Backend: `INFO:     Uvicorn running on http://127.0.0.1:8000`
    * Frontend: `Ready on http://localhost:3000`
  </Step>

  <Step title="Access the Application">
    Open your browser and navigate to:

    <CardGroup cols={3}>
      <Card title="Frontend UI" icon="browser">
        [http://localhost:3000](http://localhost:3000)
      </Card>

      <Card title="Backend API" icon="code">
        [http://localhost:8000](http://localhost:8000)
      </Card>

      <Card title="API Docs" icon="book">
        [http://localhost:8000/docs](http://localhost:8000/docs)
      </Card>
    </CardGroup>
  </Step>
</Steps>

## Generate Your First llms.txt

Now that everything is running, let's generate your first `llms.txt` file:

<Steps>
  <Step title="Open the Web Interface">
    Navigate to [http://localhost:3000](http://localhost:3000) in your browser.
  </Step>

  <Step title="Enter a Website URL">
    Enter a website URL you want to crawl. For testing, try:

    * `https://docs.python.org`
    * `https://fastapi.tiangolo.com`
    * Your own documentation site
  </Step>

  <Step title="Configure Crawl Parameters">
    Adjust the settings based on your needs:

    * **Max Pages**: Number of pages to crawl (default: 50)
    * **Description Length**: Character limit for page excerpts (default: 500)
    * **Enable Auto-Update**: Schedule periodic recrawls (optional)
    * **Recrawl Interval**: Minutes between updates (default: 360)
    * **LLM Enhancement**: AI-powered optimization (optional)
    * **Use Brightdata**: For JavaScript-heavy sites (optional)
  </Step>

  <Step title="Start Crawling">
    Click "Generate llms.txt" and watch the real-time progress in the log window.

    You'll see messages like:

    ```
    Starting crawl of https://example.com
    Crawling page 1/50...
    Crawling page 2/50...
    Found 25 pages
    Checking for .md versions of pages...
    Found 3 pages with .md versions
    ```
  </Step>

  <Step title="Get Your Results">
    Once complete, you'll receive:

    * Generated `llms.txt` content (viewable in browser)
    * Download button for the file
    * Public CDN URL for hosting
    * Copy button for quick sharing
  </Step>
</Steps>

## Understanding the WebSocket API

The backend uses WebSockets for real-time communication. Here's how the protocol works:

### Connection

```javascript theme={null}
const ws = new WebSocket('ws://localhost:8000/ws/crawl?api_key=YOUR_KEY');
```

### Send Request

<CodeGroup>
  ```json Request Payload theme={null}
  {
    "url": "https://example.com",
    "maxPages": 50,
    "descLength": 500,
    "enableAutoUpdate": false,
    "recrawlIntervalMinutes": 360,
    "llmEnhance": false,
    "useBrightdata": false
  }
  ```
</CodeGroup>

### Receive Messages

The server sends different message types:

<CodeGroup>
  ```json Log Message theme={null}
  {
    "type": "log",
    "content": "Crawling page 1/50..."
  }
  ```

  ```json Result Message theme={null}
  {
    "type": "result",
    "content": "# Example.com\n\n## Overview\n\n> Example page description...\n"
  }
  ```

  ```json URL Message theme={null}
  {
    "type": "url",
    "content": "https://pub-xxx.r2.dev/llms.txt"
  }
  ```

  ```json Error Message theme={null}
  {
    "type": "error",
    "content": "Failed to fetch page: Connection timeout"
  }
  ```
</CodeGroup>

### Implementation Example

Here's the core WebSocket handler from the backend:

<CodeGroup>
  ```python main.py theme={null}
  @app.websocket("/ws/crawl")
  async def websocket_crawl(websocket: WebSocket):
      # Validate API key
      api_key = websocket.query_params.get("api_key")
      if api_key != settings.api_key:
          await websocket.close(code=1008, reason="Unauthorized")
          return

      await websocket.accept()

      try:
          # Receive configuration
          data = await websocket.receive_text()
          payload = json.loads(data)

          url = str(payload['url'])
          max_pages = payload.get('maxPages', 50)
          desc_length = payload.get('descLength', 500)

          # Log function for real-time updates
          async def log(message: str):
              await websocket.send_json({"type": "log", "content": message})

          # Start crawling
          crawler = LLMCrawler(
              url,
              max_pages,
              desc_length,
              log,
              brightdata_enabled=payload.get('useBrightdata', False)
          )
          pages = await crawler.run()

          # Format output
          llms_txt = format_llms_txt(url, pages, md_url_map)

          # Send result
          await websocket.send_json({"type": "result", "content": llms_txt})

          # Save to storage
          hosted_url = await save_llms_txt(url, llms_txt, log)
          await websocket.send_json({"type": "url", "content": hosted_url})

      except Exception as e:
          await websocket.send_json({"type": "error", "content": str(e)})
      finally:
          await websocket.close()
  ```
</CodeGroup>

## Docker Setup (Optional)

For a simpler setup, use Docker Compose:

<Steps>
  <Step title="Configure Environment Files">
    Create `.env` files as described in steps 3-4 above.
  </Step>

  <Step title="Start Services">
    ```bash theme={null}
    docker-compose up -d
    ```

    This starts both backend and frontend:

    <CodeGroup>
      ```yaml docker-compose.yml theme={null}
      version: "3.9"

      services:
        backend:
          build: ./backend
          ports:
            - "8000:8000"
          env_file:
            - ./backend/.env
          restart: unless-stopped

        frontend:
          build: ./frontend
          ports:
            - "3000:3000"
          env_file:
            - ./frontend/.env.local
          environment:
            - NEXT_PUBLIC_WS_URL=ws://localhost:8000/ws/crawl
          restart: unless-stopped
          depends_on:
            - backend
      ```
    </CodeGroup>
  </Step>

  <Step title="Access the Application">
    Same URLs as manual setup:

    * Frontend: [http://localhost:3000](http://localhost:3000)
    * Backend: [http://localhost:8000](http://localhost:8000)
    * API Docs: [http://localhost:8000/docs](http://localhost:8000/docs)
  </Step>
</Steps>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Backend won't start - ModuleNotFoundError">
    Make sure you've activated the virtual environment and installed dependencies:

    ```bash theme={null}
    cd backend
    source venv/bin/activate
    pip install -r requirements.txt
    ```
  </Accordion>

  <Accordion title="Playwright browser not found">
    Install Playwright browsers:

    ```bash theme={null}
    playwright install chromium
    ```
  </Accordion>

  <Accordion title="WebSocket connection fails">
    Verify:

    1. Backend is running on port 8000
    2. CORS\_ORIGINS includes your frontend URL
    3. API key is configured (if required)
    4. Check browser console for error messages
  </Accordion>

  <Accordion title="R2 storage errors">
    Ensure your R2 credentials are correct:

    * Endpoint URL format: `https://<account-id>.r2.cloudflarestorage.com`
    * Access key and secret key are valid
    * Bucket exists and is accessible
    * Public domain is configured correctly
  </Accordion>

  <Accordion title="Frontend can't connect to backend">
    Check `NEXT_PUBLIC_WS_URL` in `.env.local`:

    * Should be `ws://localhost:8000/ws/crawl` for local development
    * Use `wss://` for production with HTTPS
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Configuration Guide" icon="gear" href="/guides/configuration">
    Learn about all configuration options and environment variables
  </Card>

  <Card title="API Reference" icon="code" href="/api/websocket">
    Explore the full API documentation and endpoints
  </Card>

  <Card title="Deployment" icon="cloud" href="/deployment/overview">
    Deploy to AWS with Terraform for production use
  </Card>

  <Card title="Architecture" icon="diagram" href="/architecture/overview">
    Deep dive into system architecture and components
  </Card>
</CardGroup>

<Note>
  For production deployment, see the [Deployment Guide](/deployment) which covers AWS ECS, Lambda, and infrastructure setup with Terraform.
</Note>