# llms.txt Generator

## Docs

- [Authentication](https://mintlify.wiki/Anwitht21/llmstxt/api/authentication.md): Generate short-lived JWT tokens for WebSocket authentication
- [Crawler](https://mintlify.wiki/Anwitht21/llmstxt/api/crawler.md): LLMCrawler class for crawling websites and extracting content
- [Cron Recrawl Trigger](https://mintlify.wiki/Anwitht21/llmstxt/api/cron-triggers.md): Internal endpoint for scheduled automatic recrawls
- [Database](https://mintlify.wiki/Anwitht21/llmstxt/api/database.md): Supabase database operations for site tracking and scheduling
- [Formatter](https://mintlify.wiki/Anwitht21/llmstxt/api/formatter.md): Format crawled pages into llms.txt markdown format
- [Storage](https://mintlify.wiki/Anwitht21/llmstxt/api/storage.md): R2 storage functions for uploading llms.txt files
- [Site Change Webhook](https://mintlify.wiki/Anwitht21/llmstxt/api/webhooks.md): Trigger immediate recrawl when site content changes
- [WebSocket Crawl Endpoint](https://mintlify.wiki/Anwitht21/llmstxt/api/websocket.md): Real-time website crawling via WebSocket connection
- [Data Flow](https://mintlify.wiki/Anwitht21/llmstxt/architecture/data-flow.md): Complete data flow from crawl request to llms.txt generation and storage
- [AWS Infrastructure](https://mintlify.wiki/Anwitht21/llmstxt/architecture/infrastructure.md): AWS infrastructure components, configuration, and deployment architecture
- [Architecture Overview](https://mintlify.wiki/Anwitht21/llmstxt/architecture/overview.md): High-level architecture and system design of the llms.txt Generator
- [Technology Stack](https://mintlify.wiki/Anwitht21/llmstxt/architecture/tech-stack.md): Comprehensive breakdown of technologies used in the llms.txt Generator
- [AWS Setup](https://mintlify.wiki/Anwitht21/llmstxt/deployment/aws-setup.md): Configure AWS CLI and identify network resources for deployment
- [Database & Storage Setup](https://mintlify.wiki/Anwitht21/llmstxt/deployment/database-storage.md): Configure Supabase PostgreSQL database and Cloudflare R2 object storage
- [Docker & ECS Deployment](https://mintlify.wiki/Anwitht21/llmstxt/deployment/docker-ecs.md): Build Docker image and deploy the backend API to ECS Fargate
- [Frontend Deployment](https://mintlify.wiki/Anwitht21/llmstxt/deployment/frontend.md): Deploy the Next.js frontend application to Vercel
- [Monitoring & Logs](https://mintlify.wiki/Anwitht21/llmstxt/deployment/monitoring.md): Monitor your deployment with CloudWatch logs, metrics, and alerts
- [Deployment Overview](https://mintlify.wiki/Anwitht21/llmstxt/deployment/overview.md): Understand the llms.txt Generator deployment architecture and components
- [Prerequisites](https://mintlify.wiki/Anwitht21/llmstxt/deployment/prerequisites.md): Install required tools and create service accounts for deployment
- [Terraform Infrastructure](https://mintlify.wiki/Anwitht21/llmstxt/deployment/terraform.md): Configure and deploy AWS infrastructure using Terraform
- [Auto Updates](https://mintlify.wiki/Anwitht21/llmstxt/features/auto-updates.md): Scheduled recrawls with Lambda functions, EventBridge automation, and smart change detection
- [Intelligent Crawling](https://mintlify.wiki/Anwitht21/llmstxt/features/intelligent-crawling.md): BFS traversal with automatic sitemap discovery, depth control, and smart content detection
- [LLM Enhancement](https://mintlify.wiki/Anwitht21/llmstxt/features/llm-enhancement.md): AI-powered content optimization using Grok for improved clarity and structure
- [Real-time Streaming](https://mintlify.wiki/Anwitht21/llmstxt/features/real-time-streaming.md): WebSocket streaming with live progress updates, log messages, and result delivery
- [API Usage](https://mintlify.wiki/Anwitht21/llmstxt/guides/api-usage.md): Integrate the llms.txt Generator into your applications using the WebSocket API
- [Configuration](https://mintlify.wiki/Anwitht21/llmstxt/guides/configuration.md): Complete reference for environment variables and configuration options
- [llms.txt Specification](https://mintlify.wiki/Anwitht21/llmstxt/guides/llmstxt-spec.md): Understanding the llms.txt format and how the generator implements the specification
- [Using the Web Interface](https://mintlify.wiki/Anwitht21/llmstxt/guides/web-interface.md): Learn how to generate llms.txt files using the web-based user interface
- [Introduction](https://mintlify.wiki/Anwitht21/llmstxt/introduction.md): Automatically generate and maintain llms.txt files for your website with real-time streaming and scheduled updates
- [Quick Start](https://mintlify.wiki/Anwitht21/llmstxt/quickstart.md): Get your first llms.txt file generated in 5 minutes with local development setup