Technology Stack

Comprehensive breakdown of technologies used in the llms.txt Generator

Overview

The llms.txt Generator is built with modern, production-ready technologies optimized for performance, scalability, and developer experience.

Backend Stack

Core Framework

FastAPI

Version: 0.122+

• Asynchronous Python web framework
• Built-in WebSocket support
• Automatic API documentation (OpenAPI/Swagger)
• High performance with async/await
• Type hints with Pydantic validation

Python

Version: 3.11+

• Modern async/await syntax
• Excellent library ecosystem
• Strong type hinting support
• Performance improvements in 3.11+

Web Crawling & Scraping

Playwright

• Headless browser automation
• Chromium engine for JavaScript execution
• Full page rendering support
• Network request interception
• Screenshot capabilities

BeautifulSoup4

• HTML/XML parsing
• DOM traversal and manipulation
• Content extraction
• Robust error handling

Brightdata

• Proxy service for JS-heavy sites
• Scraping Browser API
• Bypass anti-bot protections
• Global proxy network

httpx

• Async HTTP client
• HTTP/2 support
• Connection pooling
• Automatic redirects

LLM Integration (Optional)

OpenRouter

• LLM API aggregator
• Access to multiple models (Grok, GPT, Claude, etc.)
• Content enhancement and optimization
• Fallback model support

Grok 4.1-Fast

• Default enhancement model
• Fast response times
• Content summarization
• Structured output generation

Data Validation & Processing

# Key libraries
from pydantic import BaseModel, HttpUrl, Field
from dataclasses import dataclass
import hashlib  # Content hashing for change detection
import asyncio  # Async/await concurrency

Pydantic Models Example

from pydantic import BaseModel, HttpUrl

class CrawlRequest(BaseModel):
    url: HttpUrl
    maxPages: int = 50
    descLength: int = 500
    enableAutoUpdate: bool = False
    recrawlIntervalMinutes: int = 360
    llmEnhance: bool = False
    useBrightdata: bool = False

PageInfo Dataclass

from dataclasses import dataclass

@dataclass
class PageInfo:
    url: str
    title: str
    description: str
    snippet: str

Frontend Stack

Core Framework

Next.js

Version: 15+

• React framework for production
• App Router architecture
• Server and client components
• Built-in optimization
• API routes (used for proxying)

TypeScript

Version: 5+

• Type safety across the application
• Enhanced IDE support
• Reduced runtime errors
• Better refactoring experience

UI & Styling

Tailwind CSS

• Utility-first CSS framework
• Responsive design system
• Custom theme configuration
• Dark mode support
• Optimized production builds

shadcn/ui

• Re-usable component library
• Accessible components
• Customizable design tokens
• Copy-paste component architecture

Real-time Communication

// WebSocket client implementation
const ws = new WebSocket(`${WS_URL}?api_key=${API_KEY}`);

ws.onopen = () => {
  ws.send(JSON.stringify(crawlRequest));
};

ws.onmessage = (event) => {
  const message = JSON.parse(event.data);
  // Handle: log, result, url, error types
};

The frontend uses native WebSocket API for bidirectional, real-time communication with the backend.

Infrastructure Stack

AWS Services

ECS Fargate

• Serverless container orchestration
• Auto-scaling task management
• No EC2 instance management
• Pay-per-use pricing

ECR

• Docker image registry
• Vulnerability scanning
• Image lifecycle policies
• IAM-based access control

Application Load Balancer

• HTTP/HTTPS traffic distribution
• WebSocket support
• Health checks
• SSL/TLS termination

Lambda

• Scheduled recrawl execution
• Event-driven architecture
• 10-minute timeout (600s)
• 512MB memory allocation

EventBridge

• Cron-based scheduling
• Every 6 hours trigger
• Event-driven invocation
• CloudWatch integration

CloudWatch

• Centralized logging
• Metrics and alarms
• 14-day log retention
• Real-time monitoring

External Services

Supabase

PostgreSQL Database

• Managed PostgreSQL
• RESTful API
• Real-time subscriptions
• Row-level security
• Auto-generated API

Cloudflare R2

Object Storage

• S3-compatible API
• Global CDN delivery
• Zero egress fees
• Public domain URLs
• High availability

Vercel

Frontend Hosting

• Next.js optimized platform
• Edge network deployment
• Automatic HTTPS
• Preview deployments
• Built-in analytics

Terraform

Infrastructure as Code

• Declarative configuration
• Version control for infra
• Reproducible deployments
• State management
• AWS provider v5.0+

Development Tools

Testing

# Backend testing
pytest -v                    # Unit tests
pytest --cov=backend        # Coverage reports

# Frontend testing
npm run test                # Jest + React Testing Library
npm run test:e2e           # Playwright E2E tests

Code Quality

Backend

• Black: Code formatting
• Ruff: Fast linting
• mypy: Static type checking
• pytest: Testing framework

Frontend

• ESLint: Code linting
• Prettier: Code formatting
• TypeScript: Type checking
• Jest: Unit testing

Containerization

# Multi-stage Docker build for backend
FROM python:3.11-slim as builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.11-slim
RUN playwright install chromium
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Version Requirements

Component	Minimum Version	Recommended
Python	3.11	3.11+
Node.js	20.0	20.x LTS
Terraform	1.0	Latest 1.x
Docker	20.0	Latest
AWS CLI	2.0	Latest 2.x

Next Steps

Infrastructure Details

Deep dive into AWS infrastructure components and configuration

Data Flow

Understand how data flows through the system