The SEO world is buzzing with a new concept: llms.txt. It’s often compared to robots.txt, but instead of guiding Googlebot or Bingbot, it is meant to guide Large Language Models (LLMs) like ChatGPT, Claude, or Perplexity.

But does it actually work? Will Google and AI search engines ever adopt it? Or is it just another passing idea?

Let’s break it down in simple words.

What Is llms.txt?

The idea of llms.txt was introduced in September 2024 by Jeremy Howard, Co-Founder of Answer.AI.

  • It’s a plain text file, placed at the root of your website (just like robots.txt).
  • Its purpose is to tell AI models where to find clean, simplified versions of your content, often in Markdown (.md) format.
  • Instead of crawling heavy HTML pages with ads, menus, and scripts, an LLM could go directly to those clean files to get accurate information.

In short, llms.txt is supposed to act as a shortcut for AI models to read and understand your site.

Is It Effective for SEO and AI Search?

For Traditional Search Engines (Google, Bing, etc.):

Google and other search engines do not use llms.txt. They rely on robots.txt, sitemaps, structured data, and links.

For AI Search Engines (ChatGPT, Perplexity, Claude, etc.):

In theory, AI systems could use llms.txt to fetch:

  • Short, structured content
  • Key pages
  • Summaries in Markdown

This could make your content easier to pull into AI Overviews, chatbots, and generative search results.

But in reality, adoption is almost zero today.

What Does Google Say About It?

As of now (August 2025):

  • Google has not endorsed llms.txt.
  • Google’s crawling is still based on robots.txt, sitemaps, and structured data.
  • For AI Overviews (Gemini in Search), Google has never mentioned llms.txt as a signal.

Google doesn’t reject it, but it doesn’t use it either.

Who Supports and Who Doesn’t?

  • Supporters: Some documentation platforms (Mintlify, Instructor, FastHTML) and SEO tools (Yoast SEO recently added llms.txt support).
  • Skeptics: Big names in SEO like Brett Tabke (Pubcon CEO) and David Ogletree argue it’s unnecessary, since LLMs are just another kind of web crawler and should use the same system (robots.txt).
  • AI companies (Google, OpenAI, Anthropic, Perplexity, etc.) – None of them officially use llms.txt

The Harsh Reality: Are AI and Search Engines Ignoring llms.txt?

  • AI models are not using llms.txt.
  • They are still scraping and crawling real-time web pages, because that’s where fresh, updated data lives.
  • Even if you add llms.txt, it doesn’t stop or guide them today.

This has led some experts to say that AI companies are “cheating” users– they give the impression of respecting publishers, but behind the scenes, they rely on full-page crawling, not llms.txt.

So while the file is interesting, right now it’s more symbolic than practical.

Will llms.txt Be Effective in the Future?

That depends on three things:

  • Adoption by AI companies – If OpenAI, Google, or Anthropic officially support it.
  • Standardization – If llms.txt becomes part of web standards, like robots.txt.
  • Analytics – If tools appear to measure AI impressions (just like Google Search Console does for SEO).

Until then, its future is uncertain.

Final Thoughts

If you want to experiment and be future-ready, go ahead and add llms.txt

But don’t expect instant results. Google doesn’t use it, AI search engines don’t honor it, and most big players haven’t accepted it.

Real SEO still depends on the basics: technical SEO, content, links, and structured data.

Faqs

Does Google use llms.txt for SEO?

  • No. Google does not use llms.txt for crawling, indexing, or ranking. Google still follows robots.txt, sitemaps, and structured data. Adding llms.txt will not improve your SEO in Google Search.

Is llms.txt useful for AI search engines like ChatGPT or Perplexity?

  • Yes, it could help AI find clean, structured content. But in reality, most AI search engines today are not using it. They still crawl live web pages to get real-time information.

Who created llms.txt?

  • The llms.txt idea was introduced in September 2024 by Jeremy Howard, Co-Founder of Answer.AI. He wanted to make it easier for AI models to read websites without dealing with ads, menus, or heavy code.

Will llms.txt become important in the future?

  • It might. If AI companies like Google, OpenAI, or Anthropic decide to support it, llms.txt could become a standard. But right now, there is no guarantee, and it’s not widely used.

Does adding llms.txt stop AI from scraping my site?

  • No. AI search engines and models do not follow llms.txt today. They still crawl your real web pages. If you want to block them, you need to use robots.txt or other methods.

Are AI companies really ignoring llms.txt?

  • Yes. At the moment, no major AI company officially uses llms.txt. Some experts even say AI companies are “cheating” users by giving the impression of respecting site owners while still crawling full pages.

Should I add llms.txt to my website?

  • It won’t hurt to add it. You can try it to be future-ready. But don’t expect results right now-Google won’t use it for SEO, and AI companies aren’t following it yet.