What is llms.txt?
llms.txt is a plain-text file placed at the root of your website that tells AI language models who you are, what you publish, and where to find your content. It is the foundational declaration of your site’s AI identity.
Origin of the standard
llms.txt was proposed in 2024 as a community-driven standard for making websites legible to large language models. The concept was simple: just as robots.txt instructs web crawlers on access rules, llms.txt provides AI models with structured, self-declared context about a site’s content and identity.
Unlike proprietary structured data formats that require specific schema registrations, llms.txt is intentionally minimal. It uses plain Markdown syntax — headings, key-value pairs, and bullet lists — with no special encoding required. Any text editor can create one. Any server can serve one.
llms.txt is the single highest-leverage file you can add to your website for AI visibility. Every major AI platform that browses the web looks for it at yoursite.com/llms.txt before or during any content interaction with your domain.
llms.txt vs. robots.txt — what’s the difference?
robots.txt and llms.txt serve opposite purposes and should coexist on your server without conflict.
- Controls crawler access — what bots are allowed to fetch
- Uses disallow rules to block pages or directories
- Supported by all major search and AI crawlers
- Does not communicate site content or identity
- A list of restrictions
- Communicates site identity — who you are and what you publish
- Uses descriptive structure to guide AI understanding
- Read by LLM systems during reasoning, not just crawling
- Points AI to your most important and freshest content
- A declaration of context
You need both. robots.txt controls access; llms.txt controls comprehension. Neither replaces the other — in fact, a site with a well-tuned robots.txt that allows AI crawlers, combined with a rich llms.txt, represents the optimal configuration for AI visibility.
How AI Platforms Use It
Understanding what happens after an AI crawler fetches your llms.txt reveals exactly why the file’s structure and content matter so much. The path from your file to an AI’s answer passes through several distinct stages.
The crawl and index pipeline
When an AI platform’s crawler visits your domain, it checks for llms.txt at the root path before making decisions about how to index the rest of your site. The file serves as an entry point and a navigational map — it tells the crawler where to go next.
Initial discovery
The crawler fetches yoursite.com/llms.txt. If found, the file is parsed immediately. If not found, the crawler falls back to standard HTML crawling with no site-level context.
Identity extraction
The crawler reads the site name, About description, and metadata block. These values are stored as site-level facts associated with your domain — used whenever a user asks about your organization.
URL queue population
Every URL listed in your llms.txt — key pages, blog posts, documentation, Markdown files — is added to the crawler’s priority queue. These URLs are fetched before or instead of URLs discovered through standard link-following.
Content indexing
Each listed URL is fetched and indexed. If Markdown companion files are listed, they are preferred over their HTML equivalents due to higher content density and cleaner structure.
Freshness monitoring
The crawler revisits your llms.txt periodically to detect new content. Sites that update their llms.txt regularly — adding new posts, updating the modified date — are re-crawled more aggressively.
Platform-by-platform breakdown
PerplexityBot has documented support for llms.txt. It reads the file on first visit and uses the About description directly when generating answers about your site. Key pages and blog post URLs are crawled at higher priority than unprompted links.
GPT-4o’s web browsing fetches llms.txt during site exploration sessions. The URL list guides the browsing agent toward your most relevant content. The site description is factored into how the model frames its response about your domain.
Anthropic’s ClaudeBot indexes llms.txt and uses the metadata block to answer identity queries. In web-enabled sessions, Claude reads llms.txt first when asked to research or summarize your site, building a context window from the linked content.
Google’s Gemini surfaces llms.txt through its integration with Google Search infrastructure. Google-Extended crawls the file and incorporates its content alongside traditional structured data (Schema.org, Open Graph) for Gemini knowledge base population.
File Anatomy
A well-formed llms.txt has a predictable structure that every AI platform knows how to parse. Understanding each zone of the file—and what it communicates—lets you write one that works across all platforms consistently.
The five zones of a complete llms.txt
Every llms.txt should contain five distinct blocks, each serving a different purpose in the AI’s understanding of your site. The order matters — parsers read top-to-bottom and weight earlier content more heavily.
# Acme Studio ①
> About: Chicago-based branding and web design studio ②
specializing in B2B SaaS companies, est. 2018.
> Contact: hello@acmestudio.com
> Location: Chicago, IL, USA
> Modified: 2026-05-11
## Key Pages ③
- About: acmestudio.com/about
- Services: acmestudio.com/services
- Portfolio: acmestudio.com/portfolio
## Recent Posts ④
- Design Trends 2026 (2026-04-12):
acmestudio.com/blog/trends-2026
- Working with AI Tools (2026-03-20):
acmestudio.com/blog/ai-tools
## Full Content ⑤
- About (MD): acmestudio.com/about.md
- Services (MD): acmestudio.com/services.mdA single # Heading with your exact site or organization name. This is the citation name AI tools will use when referencing you.
Blockquote-style key-value pairs. The About line is the most important — it’s the sentence AI tools pull verbatim to describe you.
Your most important evergreen pages with direct URLs. Tells AI tools the core structure of your site.
Your latest content. Include publish dates inline for freshness signals. Update this section whenever you publish.
Links to .md companion files. AI crawlers fetch these for deeper, cleaner content understanding.
Syntax rules you must follow
The llms.txt format is deliberately loose, but a handful of conventions are load-bearing. Deviating from them causes parsers to misread or skip sections of your file.
One H1 heading, as the very first line
The file must begin with a single # Site Name heading. No preamble, no blank lines before it. This is the anchor AI parsers use to identify which domain the file belongs to.
Metadata uses blockquote syntax
Each metadata pair begins with > (a Markdown blockquote marker) followed by Key: Value. Parsers specifically look for this pattern to extract structured identity data.
Sections use H2 headings
Content groupings — Key Pages, Recent Posts, Documentation, Full Content — are delineated with ## Section Name headings. Don’t use H3 or H4 for section separators.
URLs are plain — no Markdown link syntax
Write URLs as plain text: yoursite.com/page, not [Page](yoursite.com/page). Many LLM parsers extract raw text and the bracketed format introduces noise that breaks URL extraction.
Your server must return llms.txt with a Content-Type: text/plain header. If it returns text/html, some parsers may attempt to render it as HTML, stripping the Markdown syntax that gives the file its structure. AI Atlas sets the correct content type automatically.
Writing Your Summary
The About line and metadata block are the most consequential sentences on your entire website for AI visibility. When an AI assistant answers “What is [your site]?”, the About line is the most likely direct source. Write it accordingly.
Crafting the About line
The About line should be a single, dense, factual sentence that a stranger could read and immediately understand exactly what your site is, does, and who it serves. Think of it as a press release lede written for a machine.
A strong About line answers five questions in one sentence: What type of entity, Who it serves, What it provides, Where it operates (if relevant), and When it was established. Not every line needs all five, but the more you include, the more AI tools can say about you accurately.
A passionate team of creatives helping brands tell their story through design.
We build websites and digital marketing solutions for businesses.
Chicago-based branding and web design studio specializing in B2B SaaS companies, founded 2018, serving clients across North America and Europe.
WordPress plugin that auto-generates llms.txt and per-page Markdown files to make WordPress sites readable and citable by AI assistants.
Metadata key-value pairs
Beyond the About line, the metadata block accepts any key-value pair that provides useful structured context about your site. While there is no enforced schema, certain keys are parsed by most AI platforms and worth including consistently.
> About: Your one-sentence site description here. ①
> Contact: hello@yoursite.com ②
> Location: City, State/Province, Country
> Founded: 2020
> Language: en ③
> License: CC BY 4.0 ④
> Modified: 2026-05-11 ⑤Required. The one sentence all AI tools pull first. Keep it under 200 characters for reliable truncation-free parsing.
Identity facts used when AI tools answer “how do I reach them?” or “where are they based?” queries.
ISO 639-1 language code. Helps multilingual AI systems route to the correct version of your content.
Declares the content license. Important for AI training dataset compliance and attribution requirements.
ISO 8601 date of when the file was last updated. Used as the primary freshness signal for recrawl scheduling.
Content Groupings
The body of your llms.txt is a navigational map of your site. The way you group and label your content determines how well AI tools understand the scope, structure, and depth of what you publish.
Structuring your sections effectively
Rather than listing all your content in one flat block, divide it into named sections that reflect the topical or functional groupings of your site. Each ## Section Name tells an AI model: “everything below this heading belongs to the same category.”
Your llms.txt sections should mirror how a human would describe your site’s structure. If your site has Products, Blog, Documentation, and Case Studies, use exactly those section names. Don’t invent generic names like “Content” or “Pages” that strip meaningful context.
The following example shows a well-organized llms.txt body for a software product site with multiple content types:
## Products
- AI Atlas Free: aiatlas.tech/free
- AI Atlas Pro: aiatlas.tech/pro
## Documentation
- Getting Started: aiatlas.tech/docs/start
- Configuration: aiatlas.tech/docs/config
- API Reference: aiatlas.tech/docs/api
- Changelog: aiatlas.tech/docs/changelog
## Guides
- AI Visibility Guide: aiatlas.tech/guides/ai-visibility
- Markdown for AI: aiatlas.tech/guides/markdown
- llms.txt Guide: aiatlas.tech/guides/llmstxt
## Blog — AI & SEO
- AI Search is Changing SEO (2026-05-01):
aiatlas.tech/blog/ai-search-seo
## Blog — WordPress
- Best AI Plugins for WordPress (2026-04-18):
aiatlas.tech/blog/ai-plugins-wordpress
## Full Content
- Getting Started (MD): aiatlas.tech/docs/start.md
- AI Visibility (MD): aiatlas.tech/guides/ai-visibility.mdFreshness signals that matter
AI platforms weight newer content more heavily when answering time-sensitive queries. Two mechanisms in llms.txt communicate freshness explicitly.
> Modified: metadata key — update this every time you add or change content(YYYY-MM-DD) after post titles in the Recent Posts sectionAn outdated llms.txt that still lists content from 18 months ago, with no recent posts and an old Modified date, signals to AI crawlers that your site is dormant. If AI Atlas isn’t updating your file automatically, set a calendar reminder to review it monthly.
Advanced Patterns
Once your baseline llms.txt is live and functioning, a set of advanced patterns can significantly increase the depth of AI comprehension of your site — without dramatically increasing file complexity.
Linking Markdown companion files
The Full Content section of your llms.txt is the single biggest quality multiplier available to you. By listing .md companion file URLs, you give AI crawlers a direct path to high-density, clean content for every important page on your site.
Think of your llms.txt as a table of contents that points to Markdown files, and Markdown files as the actual chapters. A crawler that reads your llms.txt, then follows its links to your .md files, ends up with a complete, structured, high-fidelity picture of your entire site.
For large sites, you don’t need to list every Markdown file — list the most important ones. AI crawlers will discover others through link-following once they’ve started indexing your content. Prioritize your highest-value pages: key product or service pages, cornerstone guides, and documentation index pages.
- Crawler fetches HTML and parses through structural noise
- Navigation, footer, sidebar all enter the content model
- Higher risk of misattribution and hallucination
- Slower to re-index on content updates
- Crawler fetches pure content from the Markdown file
- Zero structural noise — only your actual content
- Lower hallucination risk, more accurate citations
- Faster re-index when file timestamps change
Multi-language and multi-site patterns
Sites with multiple languages or sub-domains require a coordinated llms.txt strategy. There is no single official pattern, but the following approach is widely adopted and correctly parsed by major AI platforms.
# Acme Studio
> About: Chicago-based branding studio, B2B SaaS, est. 2018.
> Language: en
> Modified: 2026-05-11
## Key Pages
- About (EN): acmestudio.com/about
- Services (EN): acmestudio.com/services
## Localized Versions
- French site: acmestudio.com/fr/llms.txt
- German site: acmestudio.com/de/llms.txtEach localized sub-path or sub-domain should host its own llms.txt — written in that language, with its own About description and content listings. The root llms.txt then references these per-locale files under a Localized Versions section, giving AI crawlers the full map from a single entry point.
Listing English and French content together in one llms.txt with no language separation causes AI tools to conflate the two audiences.
Root llms.txt in English, with a Localized Versions section pointing to /fr/llms.txt and /de/llms.txt — each fully self-contained.
WordPress & Automation
Manually maintaining a llms.txt on a WordPress site is a losing battle. Every new post, updated page, or removed content requires a file edit. AI Atlas was built specifically to eliminate this maintenance burden entirely.
Using AI Atlas to automate llms.txt
AI Atlas hooks into WordPress’s core content lifecycle — publish, update, trash — and regenerates your llms.txt automatically any time your content changes. Your AI footprint is always a live, accurate reflection of your site’s current state.
Auto-generates llms.txt with your site name, About description (pulled from tagline), Contact, all published pages, and your 10 most recent posts. File updates on every content save. Serves with correct text/plain content type.
Everything in Free, plus: custom About line editor, unlimited posts, custom post type support, per-section groupings matching your site taxonomy, inline date injection, Markdown file linking in Full Content section, and multi-language llms.txt generation per locale.
Validation checklist
After installing AI Atlas and activating llms.txt generation, work through this checklist to confirm your file is correct, live, and being picked up by AI platforms.
A live llms.txt is only as good as its maintenance cadence. With AI Atlas, every content update regenerates the file automatically — but review your About line and metadata block quarterly to ensure they still accurately represent your site as it evolves. AI visibility is a practice, not a one-time configuration.
Ready to be found by AI?
Track your brand across every major AI and gives you the tools to act on it. See when you're mentioned, how you're described, where competitors are beating you, and exactly what to fix to show up more.