What robots.txt does
robots.txt lives at yourdomain.com/robots.txt. Crawlers read it before they crawl, and each rule names a crawler and the paths it may not crawl. Google says the file exists mainly to manage crawler traffic. It keeps bots away from pages that waste crawl time, like internal search results or endless filter combinations.
What robots.txt does not do
- Controls what crawlers may fetch
- Cannot keep a URL out of results
- Google cannot read a blocked page
- Use it for low-value crawl paths
- Controls what appears in results
- Keeps the page out of the index
- Google must crawl it to see the tag
- Use it for pages people should not find
Do not combine the two on one page. If robots.txt blocks a page, Google cannot crawl it and never sees its noindex tag. Google says a blocked URL can still appear in results, without a description, when other sites link to it.
A safe robots.txt for a local business site
Most small sites need very little. Allow everything, block only what wastes crawl time and point to your sitemap. Never block the CSS and JavaScript files your pages need, or Google cannot render them properly.
The rules also apply to AI crawlers. If you want ChatGPT or Perplexity to cite your pages, do not block their search crawlers.
The mistakes we see most
- A staging rule
Disallow: /left in place after launch - Service pages blocked by an overly broad rule
- robots.txt used to hide private pages, which it cannot do
- Blocked scripts that stop Google rendering the page
After any change, open the robots.txt report in Search Console and test your key URLs. See why pages are not indexed if pages go missing. Our technical SEO work checks these files on every site we take over.