[← All posts](https://www.crawlreadiness.com/blog)

September 15, 2026 · 8 min read

# What is llms.txt? The complete 2026 guide

llms.txt is a proposed standard for helping AI systems understand your website. Here's what it is, why it matters, and how to add one in under 10 minutes.

## The 30-second answer

**llms.txt** is a plain-text file you place at the root of your website (like `yourdomain.com/llms.txt`) that tells AI assistants — ChatGPT, Claude, Perplexity, and others — which pages on your site are worth reading and how they should interpret them. Think of it as `robots.txt` and `sitemap.xml` had a baby, but for the AI era.

The proposal was introduced by Jeremy Howard in September 2024 at [llmstxt.org](https://llmstxt.org/) and has since been adopted by companies including Anthropic, Cloudflare, Perplexity, Zapier, Shopify, and many others.

## Why does llms.txt exist?

AI assistants have a fundamental problem when they visit your website: your homepage is designed for humans. It has navigation bars, cookie banners, marketing copy, and pretty pictures — and often relies on JavaScript to render meaningful content. That's fine for people. It's terrible for a language model trying to answer a customer's question in real time.

llms.txt solves this by giving the AI a hand-curated map of your site in the format it actually prefers: clean Markdown, with links and short descriptions of what each page covers. No CSS. No hydration mismatches. Just structured content.

## What does a good llms.txt file look like?

The format is simple Markdown, with a specific structure:

```
# Your Site Name

> Short description of what your site is about.

Optional deeper context in a paragraph or two.

## Docs
- [Getting started](https://yoursite.com/docs/getting-started): Set up your first project.
- [API reference](https://yoursite.com/docs/api): Full API docs.

## Blog
- [Announcement post](https://yoursite.com/blog/launch): Why we built this.

## Company
- [About](https://yoursite.com/about): Who we are.
- [Pricing](https://yoursite.com/pricing): How it works.
```

The `# heading` is your site name. The `> blockquote` is a short one-line summary. Each `## heading` is a section, and each bullet is a`[Title](URL): description`.

## Do AI systems actually use it?

As of mid-2026: **partially, and increasingly.** Here's where things stand:

- **Anthropic (Claude)** and **Perplexity** read llms.txt when available.
- **OpenAI (ChatGPT)** hasn't officially confirmed usage but many sites report that adding llms.txt improved citation quality in ChatGPT answers.
- **Google AI Overviews** uses your regular sitemap and content — llms.txt isn't official for them yet, but Google-Extended (Gemini training) may consult it.
- Many second-tier AI crawlers (Kagi, Andi, You.com) explicitly reference llms.txt in their docs.

Even if a specific AI system doesn't read it today, adding llms.txt is nearly free — a few minutes of work — and gives you a machine-friendly summary of your site that will be used more as the standard matures.

## How to add llms.txt to your site

1. **Generate the file.** You can hand-write it or use our free [llms.txt Generator](https://www.crawlreadiness.com/tools/llms-txt) — it crawls your site, picks the most useful pages, and produces a properly formatted file in seconds.
2. **Edit the descriptions.** The auto-generated descriptions come from your meta tags and OG data. Rewrite them in your voice — this is what an AI will "see" when it summarizes your site.
3. **Upload it to your site root.** On WordPress: via File Manager to public_html. On Vercel/Netlify: drop it in `/public`. On Shopify or Squarespace: this is trickier — see the platform-specific instructions in our tool.
4. **Verify it works.** Visit `yourdomain.com/llms.txt` in a browser. You should see plain text, not an HTML page.
5. **Add a companion robots.txt entry.** Add `Allow: /llms.txt` under `User-agent: *` in your robots.txt so crawlers know it exists.

## Common mistakes to avoid

- **Skipping descriptions.** An llms.txt with just links is missing the point — the descriptions are what AI uses to decide which pages are relevant to a query.
- **Including too many pages.** This isn't your sitemap. Include the 10-30 pages that best represent what your site offers. Everything else is noise.
- **Listing broken or gated URLs.** If an AI can't actually fetch the page, listing it in llms.txt hurts more than helps.
- **Forgetting to update it.** Regenerate llms.txt when you launch major new sections or restructure your site.

## llms.txt vs llms-full.txt

There's a secondary standard called `llms-full.txt` that includes the full Markdown content of every page inline. It's useful for documentation-heavy sites where an AI might want to answer questions without fetching individual pages. Most sites should start with just `llms.txt` — it's simpler and covers the 80% case.

## Bottom line

llms.txt is a low-effort, high-signal addition to your site's AI-readiness stack. It won't fix a site that AI crawlers can't reach at all (that's what our [free readiness check](https://www.crawlreadiness.com/) is for), but for sites that are already crawlable, it meaningfully improves how AI assistants describe and cite you.

Generate yours in about 30 seconds using our [free llms.txt Generator](https://www.crawlreadiness.com/tools/llms-txt). It includes a companion robots.txt snippet and platform-specific install instructions for WordPress, Shopify, Squarespace, Wix, Webflow, Vercel, Netlify, and Cloudflare Pages.

### Try Crawl Readiness free

Check whether ChatGPT, Claude, Perplexity, and 50+ other AI crawlers can access your website — no signup for the basics.

[Run a free readiness check →](https://www.crawlreadiness.com/)

---

Source: https://www.crawlreadiness.com/blog/what-is-llms-txt