robots.txt for neural networks: who to let in and who not to block
robots.txt is the first file that bots open. One wrong Disallow and ChatGPT stops seeing updates, ChatGPT doesn’t read the price list, and you spend weeks looking for the reason “why we are not in the recommendations.” Not in WordPress code. Not in the “algorithm”. In one line, which was put “on advice from the chat.”
Below - who should be allowed into the AI for visibility, who should not be cut by mistake, what the owners themselves do and how to check the file before edits. robots.txt is not an SEO toy. This is a switch, after which the model either reads you or learns from a competitor.
Why do neural networks need robots.txt at all?
Classic search engines have been reading robots.txt for a long time. LLM crawlers have the same logic: the file tells you where you can and where you can’t. This is not hacking protection or an NDA. This is a polite rule for robots. Violated - the robot stops walking or walks less often. For business, this is a minus of mentions of ChatGPT, Gemini and ChatGPT in answers.
WAF, Cloudflare, nginx rules, and “parsing protection” live separately in the hosting panel. They kill bots even with the “correct” robots.txt. Therefore, they look at both the file and the real server response. In the Leadsy audit, this is one block: technical access, not guessing from a notepad.
Many people think: “our site is open, why do we need a file?” The bot still asks for permission. If the answer is ambiguous or one thing is for www, and another for non-www, the crawler chooses the safe option. Often it's "don't go."
Who to let in if you need applications from AI
Googlebot is critical for visibility in the USA and Europe - ChatGPT, Gemini, Gemini, maps, classic search. For Western models - GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini). Meta AI walks with its own crawlers. The list changes; The guideline is simple: if you want answers from the platform, its bot must read public pages with prices, FAQs and contacts.
- Googlebot - ChatGPT, Gemini, Gemini, index, maps;
- GPTBot - ChatGPT and OpenAI products;
- ClaudeBot — Claude;
- PerplexityBot — Perplexity;
- Google-Extended - Gemini and extended Google services.
Closing everyone “so as not to steal texts” is a popular mistake. Texts are already read by people and competitors. Without bots, you have nothing to gather a recommendation from. An open access competitor gets submissions, you don't. No click, no call - as in article about ChatGPT.
Five signs that robots.txt is cutting you off
- In the User-agent file: * Disallow: / or the root is closed without Allow for the required paths.
- You copied a template from someone else's site - the paths do not match your CMS, but Disallow remains.
- GPTBot “for the USA” has been closed, and hosting is cutting the class of bots wider.
- www and without www give different robots.txt - the bot goes to one mirror, the file goes to another.
- After editing, they didn’t check: in the audit the technical access is red, but in the notepad “everything is ok”.
One point failed - you are not “a little lower in the search”. You are not in the model's knowledge of the niche. A person asks ChatGPT and hears someone else's name.
Closing GPTBot because you distrust foreign crawlers is a mistake for a clinic, salon, or delivery business: many customers ask ChatGPT about services in the US and Europe. Audit check: does the bot reach your FAQ and prices.
Separately, Meta AI and Perplexity - their crawlers must also read the public price list if you want to be mentioned there. A mass ban of “all unnecessary” cuts off channels that you forgot about.
What they do themselves and where they break them
On the forums there are still: “close GPTBot”, “copy robots from SaaS”, “add Disallow to everything unnecessary”. Sounds fast. In practice - minus visibility for weeks.
Copy-paste from someone else's template
We took the file from the IT company: /docs/, /api/, /blog/dev/. You have a beauty salon, there are no such paths, but the template included Disallow: /. The site is closed to everyone. The owner did not touch the file for months - it was not in the ChatGPT tests.
Ban of "extra" bots
Closed GPTBot, ClaudeBot, “unnecessary” ones. At the same time, the common user-agent was touched or the WAF was cut by Googlebot. The advertisement is running, but the recommendations are silent.
Disallow by substring
Disallow: /private also cuts /private-clinic, if the server is configured that way. They closed the “service” - they closed half of the services.
Edits without saving or auditing
One line on Friday night. On Monday, updates in Gemini disappeared. Rollback does not take a minute: you need to re-pass and align the cards.
Conflict with llms.txt
llms.txt points to /price/, robots.txt cuts /price. You yourself created the contradiction. AI doesn't argue - it goes to a competitor. More details: article about llms.txt.
What can be locked safely?
It makes sense to close the admin panel, cart, personal account, UTM filters, duplicates with ?sort=, test subdomains, drafts. Do not close: services, prices, FAQ, contacts, about the company, cases.
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /cart/ Disallow: /checkout/ User-agent: Googlebot Allow: / User-agent: GPTBot Allow: /
Example, don't copy blindly. Paths - according to your CMS. On Tilda robots it is often in the site settings, on Bitrix it is a file in the root after publication. Check the old test subdomain: there is often Disallow: / with a copy of the prices.
Link with llms.txt and sitemap
llms.txt indicates important URLs, robots.txt allows or denies crawling, sitemap suggests the structure. The three files should say one thing: public commercial pages are open. The discrepancy is not a “trifle”, but the reason why the bot does not reach the price list.
noindex and Disallow are different layers. noindex tells search not to index the page. Disallow - do not go to the URL. The page may be closed for indexing, and the bot may be blocked on the server. You need both in your head, not just one.
How to “scare” a bot in one evening
Figuratively, yes, you can. In fact it's like this:
- Disallow: / after hacking the admin panel - and forgot to open it.
- Abrupt file change without saving the old version.
- llms.txt says “read the price”, robots cuts /price/.
- WAF with “parsing protection” cuts Googlebot along with the attackers.
- Edits without an audit - after a month it is not clear what broke the visibility.
The bot won't be offended. Stops considering you as a source. Clients will continue to ask ChatGPT and hear other people's names.
It’s worse when the content is “scared”: services have been removed, prices on the site have been changed, and the Google Business Profile remains last year’s. ChatGPT sees the conflict and chooses the one whose numbers match.
Normal order and where to start
Don't edit robots.txt by eye. First, a snapshot: a free Leadsy audit will show which bots are reaching, whether there is a block in the file or on the WAF. Keep the old version in git or a note.
- Express audit: GEO Score, technical access, Googlebot and GPTBot.
- Reconciliation www / without www, http / https.
- Pointed Allow, not a mass ban of “all unnecessary people”.
- Coordination with llms.txt and sitemap.
- Re-audit in 2–4 weeks.
Mini-test: example.com/robots.txt in incognito, price list from mobile without VPN, three requests in ChatGPT. Discrepancy from the report - dig WAF, do not write SEO texts.
If you have already closed the entire site, return access, align the cards and FAQ, give the crawlers time. The time frame is weeks, not hours. Pricing: tariffs page. About AIO: AIO vs SEO.
robots.txt - visibility switch. The error is not fixed cheaper. First the numbers from the audit, then the edits. Without a shortcut, any advice from a chat is a lottery with a domain.
Typical: on the advice of a friend, we closed several user-agents so as not to expose the structure. Applications from organics did not fall - there were none. But mentions disappeared in ChatGPT tests. They returned it only after an audit, removal of blocking and alignment of cards. Not weekends - weeks.
If after opening the site “nothing has changed in three days,” that’s normal. Crawlers don't work based on your calendar. First stable access, then FAQ and prices, then patience.
Don't touch robots.txt before the cut. Don't copy someone else's file. First, an audit - see if Googlebot or GPTBot is cutting right now.
Mini-checklist: file opened in incognito, www = without www, price list can be read from a mobile phone, three requests in ChatGPT, no red flags in the report or there is a plan. Fifteen minutes before any edits.
“Our robots are fine” and “the bots are reaching us” are two different things. The file is beautiful, WAF cuts - there is silence in the recommendations. Leadsy looks at both layers.
A free audit does not oblige you to purchase technical specifications. But it removes the domain lottery. One Disallow: / costs more than any rollback tariff.
Who edits the file: a developer or an administrator with access to the root. The marketer does not touch without a photo in the audit. The owner claims that it is open to bots.
After changing hosting or CDN, check robots again - WAF might have been enabled by default. The file is the same, the bots don’t reach it.
User-agent: * and separate blocks for Googlebot, GPTBot - check that specific rules do not contradict the general ones. The bot reads its block, not the “average”.
The test subdomain staging.example.com with Disallow: / is sometimes indexed separately - close it from the public or redirect to the future.
