How to set up llms.txt, and why one file is not enough
llms.txt is a text file in the root of the site where you list which pages matter to language models. The idea is simple: help the bot stop guessing. In practice one file solves little, and one wrong path means you sent the AI assistant to the wrong place yourself.
Below: how llms.txt is built, what to put in it, the usual mistakes when you set it up yourself, and why visibility sometimes drops after a "quick fix" instead of rising.
What llms.txt is, and why it exists
The format was proposed as an analogue of robots.txt, for LLM crawlers. The file usually contains:
- a short description of the company or project;
- a list of URLs with notes: services, prices, FAQ, contacts;
- sometimes links to markdown versions of pages, or a sitemap for models.
The file does not guarantee a recommendation. It lowers the chance that the bot grabs a random page: a 2019 blog post, an expired promo, a draft in a test folder.
Without llms.txt a model can still find you. With a broken llms.txt it can "learn" from junk faster than with no file at all.
Who reads the file: GPTBot from OpenAI, ClaudeBot, PerplexityBot, and other LLM crawlers. Yandex still leans on its own index and Yandex Neuro, and the practice overlaps: a clean site structure helps everyone. The file does not replace indexing in Yandex. It lowers the chance that a Western model grabs the wrong page when it answers a Russian-speaking user.
Do not mix up llms.txt with llms-full.txt and GitHub mirrors. Take the spec from the official format repository, and write the contents for your own domain. Templates from articles often include author and version fields you do not need. They only add noise.
Five signs of a broken llms.txt
- Paths from someone else's template lead to 404s on your domain.
- 50 or more URLs, and the model cannot tell what matters.
- The price list is a PDF. The bot sees a link and does not see the numbers.
- robots.txt blocks pages listed in the file. You are in conflict with yourself.
- The file was not updated after service URLs changed. Dead links.
A broken index is worse than no index: the bot visits, finds junk, and records the instability.
A minimal working example
A structure with no claim to be a standard. There is no single standard yet:
# Company name > Short: what you do, the city, who it is for. ## Services and prices - https://example.com/services/ - price list and timelines - https://example.com/faq/ - answers for clients ## About - https://example.com/about/ - licenses, team ## Contact - https://example.com/contacts/
Every URL has to open, return 200, and match what is on the site and on the maps. A redirect to another domain, http instead of https, a typo in the path: that is already an error.
Comments after a URL are optional, and they help: "price list current as of May", "clinic license", "FAQ for booking". One line of context lowers the chance that the model uses the page for the wrong job.
Why one file is nowhere near enough
llms.txt does not replace:
- normal commercial content on the pages;
- Schema.org and an FAQ with real questions;
- matching listings in Yandex Business and 2GIS;
- open access for the bots you need in robots.txt;
- reviews and mentions outside the site.
The file is an index. If the index leads into an empty room, the model will not recommend you. A lot of owners spend an evening on llms.txt and never touch the prices on the site. In the answer, Alice names the business whose prices and address match everywhere.
The wider AIO picture: AIO vs SEO. Alice: Alice recommendations.
Where the file lives: only /llms.txt in the domain root, not in a subfolder, not renamed to txt.pdf. For subdomains blog.example.com and example.com, use separate files if you want both visible. On Tilda and other builders you sometimes need a separate "file in the root" block. Otherwise the editor saves a page, not a text resource.
Update llms.txt when the price list, promos, or service URLs change. An old file with dead links is worse than no file: the bot visits, finds 404s, and records the instability. A check against the sitemap once a quarter is a normal minimum for a live business.
On a one-page landing the file is almost pointless. The bot already sees only that page. On a multi-page site with no llms.txt, the model more often grabs the blog, a press release, or an expired promo. You never told it what matters.
Markdown versions of pages help if they match the HTML and contain the prices. A file that links to empty markdown with no numbers is an index into an empty room again.
Mistakes when you set it up yourself
A paste from another site
You downloaded llms.txt from a SaaS template. Paths like /docs/, /api/, and /pricing/ 404 on your domain. The bot records junk. Trust in the domain does not grow.
You listed everything
50 URLs, including utility pages, tags, and city duplicates. The model cannot tell what matters. Five strong pages beat fifty noisy ones.
You forgot https and the slash
example.com/services and example.com/services/ look the same to you. To a crawler they can be different. Take the canonical address from the sitemap.
You published it and never checked
The file is there, but nginx returns 404 on /.well-known/, or a cache serves the old version. Check: open it in a private window, and run the URL through a Leadsy audit.
You blocked the bots after publishing
llms.txt says "read us". robots.txt says "no". The AI assistant does not argue. It leaves. That is how visibility gets scared off in one evening.
A version for the English site
Two languages means two sets of URLs in one file, or two files on different domains. Mix Russian and English with no labels and the model confuses the geography and the currency. For a Russian company with an English version, label the sections.
Autogeneration from the sitemap
A script dumped 500 URLs into llms.txt. It looks tidy. It is junk. A manual selection beats "let the robot sort it out".
Order of work if you do it yourself
- Audit before edits. Find out whether bots see you, and what is already broken.
- A list of 5 to 10 canonical URLs: services, prices, FAQ, about, contacts.
- Check each URL by hand and in the report.
- Publish /llms.txt in the root, no BOM, UTF-8.
- Line it up with robots.txt: the user-agents you need are not in Disallow.
- Audit again in a week, not sooner. Give the crawlers time.
If the express audit is red on technical access or schema, do that first. llms.txt is the second stage, not the first.
When it is better to order it
No developer, a builder site with messy URLs, several subdomains, an old WordPress full of duplicates: a homemade file will cost more to repair than to commission. One wrong Disallow in robots.txt cancels a tidy llms.txt.
Leadsy does a full audit with an action plan: which URLs belong in llms.txt, what to fix in schema, what to open for bots. Done-for-you implementation if nobody can put the file on the server.
Where to start today
Do not copy someone else's llms.txt. Do not publish the file before an audit. Do not block bots "just in case".
Start with a snapshot in the form below. Then 5 to 10 URLs with live content. Check it against robots.txt. Audit again in a week, not sooner.
Example: a dental clinic listed five URLs, and the price list was a PDF. The bot saw a link and did not see the prices. Alice named the clinic with prices in HTML. They moved the price list onto a page. The shift happened without making the file longer.
Typical case: an evening on llms.txt, and the prices on the site were left alone. In the answer, Alice names the business whose numbers match in HTML and on the maps. The file does not replace a price list.
On WordPress: FTP, or a static file in the root. On Bitrix: the public part. On Tilda: a zero block as a document. One check: the text opens by a direct link in a private window, not "we uploaded it in the admin".
UTF-8 with no BOM. If a 404 HTML page is served instead of text, the bots leave and trust drops.
Priority if you have no developer: (1) audit, (2) the phone and address on the site match the maps, (3) prices and FAQ in HTML, (4) robots.txt, (5) schema, (6) llms.txt, (7) another snapshot. Jumping to step six without steps 2 to 4 is the most common failure.
If the file is already up and traffic from AI answers is zero, do not delete it in a panic. Find what breaks the picture harder. Deleting it with no diagnosis leaves the holes, minus one index.
llms.txt strengthens what is already assembled. A file will not save an empty site. Pricing: the pricing page. Questions: FAQ.
A free audit shows whether the file exists, whether bots read the pages in it, and what to fix earlier. Without a snapshot, one evening with a broken file can cost a month of visibility.
Who owns llms.txt in the company: the developer places the file, marketing supplies the URL list, the owner approves the price list and the services. One person "throwing links in" with no check is the usual failure.
Do not delete the file in a panic because it "did not work in a week". Listings, schema, and robots first. Then look at the index again.
Hosting with a WAF, separately: an allowlist for LLM crawlers matters more than a new paragraph on the homepage. You can see it in the audit under technical access.
Typical case: llms.txt went up, and the price list in HTML was forgotten. Alice names the clinic with prices on the page. The file did not replace the content. It only pointed at where to look.
No more than 10 to 15 lines in the file for a multi-page site. The rest belongs in the sitemap. Otherwise you blur what matters.
Tied to schema and the FAQ: the index should lead to pages with facts, not to empty shells.