What Is LLMs.txt & Should You Use It?

What Is LLMs.txt?

The llms.txt file is a proposed standard meant to help large language models (LLMs) better understand and use content from websites. 

Here’s the official specification:

The idea is pretty straightforward: Instead of letting AI crawlers wander around your site, you give them a curated list of your most important content. To tell AI what content on your site it should actually pay attention to.

We already have standards like robots.txt and sitemaps to help search engines navigate websites more efficiently. What’s different about llms.txt is that it’s specifically designed for AI models that might use your content to answer questions or generate responses for users.

There’s also speculation that implementing llms.txt could give websites more visibility in AI-generated responses and potentially drive more referral traffic. 

Before we unpack llms.txt and determine whether it’s worth implementing, let’s understand why there’s even a need to create another web standard.

What Problem Is LLMs.txt Trying to Solve?

llms.txt is designed to help AI crawlers browse websites more effectively. Right now, these crawlers run into two major challenges:

• Modern websites are hard to read. Most AI crawlers can only read your pages’ basic HTML—not content that gets loaded by JavaScript. That means llms.txt provides a clear, structured format that helps AI crawlers quickly digest the information.
• Most websites have an overabundance of information. When AI crawlers visit your website, they don’t necessarily know what’s important. If they spend time scraping pages that aren’t useful (like older blog posts), they may generate responses based on suboptimal information—llms.txt helps solve this problem.

llms.txtmay also reduce the inefficiencies in large language model training. 

There’s a massive computational cost involved in training LLMs. With llms.txt guidance, LLMs are less likely to waste resources on irrelevant content.

How Are LLMs.txt Files Structured?

According to the proposed standard, llms.txt files should be structured and formatted in Markdown.

Markdown is a lightweight markup language that uses plain text formatting syntax to create structured documents. (It’s the same format developers use in GitHub README files, and it’s easily parseable by AI systems.)

Some common Markdown elements you’ll use in your llms.txt file include:

• # for the H1 heading, ## for H2s, ### for H3s, and so on
• > for blockquotes to highlight important descriptions
• – or * for bullet points in unordered lists
• [text](url) for hyperlinks to your content
• : for adding descriptions next to links to help explain what they lead to 
• “` for code blocks when sharing technical examples

The official llms.txt specification provides a very basic example of how your file might look. But if your website is large or complex, you might want to add more structure—using H3s and H4s to create subsections, incorporating tables to organize data, or including code snippets to demonstrate API use.

There’s nothing wrong with that. Markdown files are completely readable by AI crawlers, so you’re safe as long as you’re using valid syntax. The additional structure might actually give AI crawlers more context.

Here’s a simple example following the basic specification:

Are Brands Using the LLMs.txt Standard?

Yes, some SaaS and developer-focused companies are already using llms.txt files on their websites. 

However, the overall adoption is pretty niche. According to NerdyData, only 951 domains (a tiny fraction of the web) had published an llms.txt file as of July 2025.

Here are some example companies:

Notice how differently each company approaches their llms.txt file. They each use a different structure. 

There’s nothing wrong with this. As long as they’re using valid Markdown, the file is machine-readable and can be easily processed by AI systems.

Also, none of these companies has a file that’s focused on their website as a whole. That’s a personal choice they made. You can create a file that’s focused on your entire site or just one specific section.

Should You Use LLMs.txt on Your Site?

Using llms.txt is probably not worth your time right now, unless you’re just curious and want to experiment.

llms.txt is currently just a proposed standard rather than something that’s actually being used by the major AI companies. 

None of the LLM companies like OpenAI, Google, or Anthropic have officially said they’re following these files when they crawl websites.

Google’s John Mueller also confirmed this on Bluesky:

That said, there are some interesting signals.

For example, Anthropic has published an llms.txt file on their own website. That doesn’t mean their AI crawler is actually using these files—but it suggests they’re probably at least open to the idea.

We’re still in the early speculation phase where people are implementing the file and hoping it might become useful someday.

Is Semrush going to implement this file?

We’ve implemented llms.txt on one of our sister sites, Search Engine Land, to see whether it offers any meaningful advantages in terms of AI visibility and traffic. You can check out the file here if you’re curious.

We’ll monitor the results over the coming months and update this article with our findings.

If you also want to experiment with llms.txt on your own site, below are step-by-step instructions on how to implement it.

How to Create an LLMs.txt File (Step by Step)

This is technical, so it’s best to involve a developer in the process as you follow these three steps:

1. Decide What Content You Want to Feature

Before creating a file, determine which pages or sections of your website should be highlighted for AI crawlers.

Let’s say you want to create an llms.txt file for your whole website. At the very least, consider your:

• Product or service pages
• Up-to-date blog posts
• Pricing page
• About us page
• Contact page

These are typically the pages that will give AI a good idea of what your business does and how you help customers.

2. Create the File

Open a text editor like Notepad or Visual Studio Code and create a new file named llms.txt.

You need to format the file using Markdown. Again, developers are helpful for creating the file.

Here’s how the file’s structure could look:

3. Upload the File to Your Website

Place your completed file in the right location so AI crawlers can theoretically find it.

The exact location depends on the scope of your llms.txt file:

• If it covers your entire website, upload it to your root directory (i.e., “https://[yourdomain].com”) so it’s accessible at “https://[yourdomain.com]/llms.txt”
• If the file is specifically about documentation, place it in the corresponding subdirectory (e.g., “https://[docs.yourdomain.com]/llms.txt)

You’ll need a developer’s help to actually upload the file. This file has to be placed on your server—usually through your web hosting control panel, such as cPanel.

Log in to your hosting provider and then navigate to cPanel > File Manager.

Then go to the correct directory. If your llms.txt file is meant for the whole site, go to the public_html/ folder. (That’s the root directory for most domains.)

If it’s for a subdomain like “https://[docs.yourdomain.com],” navigate to the folder assigned to that subdomain—often named /docs/ or similar.

Upload your llms.txt file there and save the changes.

Once the file is uploaded, confirm everything is working by opening a new tab and visiting the URL directly.

You can also run a quick audit of your website in Semrush’s Site Audit tool to confirm that your llms.txt is being picked up correctly.

Also, don’t forget to keep the file updated over time. Regularly review the links to remove outdated pages. And add links to the new content you add to the website.