Deployment Overview

Understand the llms.txt Generator deployment architecture and components

Architecture Overview

The llms.txt Generator uses a modern serverless and containerized architecture deployed on AWS, with managed services for database and storage.

Key Components

Backend API

FastAPI application running on ECS Fargate behind an Application Load Balancer

Database

Supabase PostgreSQL for tracking crawl sites and metadata

Storage

Cloudflare R2 for storing generated llms.txt files

Frontend

Next.js application deployed on Vercel

Architecture Diagram

Infrastructure Components

AWS Resources

Compute (ECS Fargate)

• ECS Cluster: Container orchestration
• Task Definition: 0.5 vCPU, 1GB RAM
• Service: 1 running task with auto-healing
• ECR Repository: Docker image registry

Networking

• Application Load Balancer: Routes HTTP/HTTPS traffic
• Target Group: Health checks on /health endpoint
• Security Groups: ALB (80/443) → ECS Tasks (8000)
• VPC & Subnets: Requires 2+ subnets in different AZs

Automation

• Lambda Function: Triggers recrawl endpoint
• EventBridge Rule: Cron schedule (every 6 hours)
• S3 Bucket: Lambda deployment package storage

Monitoring

• CloudWatch Logs: Application and Lambda logs (14-day retention)
• CloudWatch Metrics: CPU, Memory, Response Time
• CloudWatch Alarms: 10 alarms for critical metrics
• SNS Topic: Email alerts for incidents

Security

• IAM Roles: Least-privilege access for ECS and Lambda
• ACM Certificate: SSL/TLS for HTTPS
• Secrets: Environment variables in task definitions

External Services

Supabase

• PostgreSQL database
• Free tier supported
• Stores site metadata and crawl history

Cloudflare R2

• S3-compatible object storage
• Public URL access
• Stores generated llms.txt files

Vercel

• Frontend hosting
• Automatic deployments from Git
• Edge network CDN

Brightdata (Optional)

• JavaScript-heavy site scraping
• Scraping Browser API
• Pay-per-use pricing

Deployment Flow

1. Prerequisites Setup

   Install required tools (AWS CLI, Terraform, Docker) and create accounts (AWS, Supabase, Cloudflare)

2. AWS Configuration

   Configure AWS CLI and identify VPC/subnet resources for ECS deployment

3. Database & Storage

   Set up Supabase database with migrations and configure Cloudflare R2 bucket

4. Terraform Variables

   Configure terraform.tfvars with all service credentials and settings

5. Infrastructure Deployment

   Run terraform apply to create all AWS resources (ECR, ECS, ALB, Lambda, monitoring)

6. Docker Build & Deploy

   Build Docker image, push to ECR, and deploy to ECS

7. Frontend Deployment

   Deploy Next.js frontend to Vercel with backend API URL

8. DNS & SSL

   Configure custom domain and SSL certificate (optional)

Resource Requirements

Compute Resources

Component	Specification	Cost Estimate
ECS Fargate Task	0.5 vCPU, 1GB RAM	~$15/month
Application Load Balancer	Standard ALB	~$16/month
Lambda Function	512MB, runs every 6 hours	Less than $1/month
CloudWatch Logs	14-day retention	~$2/month

Total AWS infrastructure costs: approximately $35-40/month for production deployment.

External Services

• Supabase: Free tier (500MB database, 1GB storage)
• Cloudflare R2: Free tier (10GB storage, 1M Class A operations)
• Vercel: Free tier (100GB bandwidth)
• Brightdata: Pay-per-use (optional, ~$0.001 per page)

Deployment Time

Expected time to complete full deployment:

• Prerequisites & Account Setup: 30-60 minutes (one-time)
• Terraform Infrastructure: 5-10 minutes
• Docker Build & Push: 5-10 minutes
• ECS Service Deployment: 3-5 minutes
• Frontend Deployment: 2-3 minutes

First-time deployments may take longer due to account setup, DNS propagation, and SSL certificate validation.

Next Steps

Prerequisites

Install required tools and create service accounts

AWS Setup

Configure AWS CLI and identify network resources

Database & Storage

Set up Supabase and Cloudflare R2

Terraform Deployment

Deploy infrastructure with Terraform