What is robots.txt?

Why it matters for your business

Every website can have a robots.txt file at yourdomain.com/robots.txt. Search engines and many AI crawlers read it before visiting your pages. A small mistake in it can block your entire site from search engines.

robots.txt controls crawling, not indexing. A page blocked by robots.txt can sometimes still appear in results if other sites link to it, just without a description. To keep a page out of search results, use a “noindex” tag instead, and let crawlers reach the page so they can see it.

Example

This robots.txt allows everything except an admin area, and points crawlers to the sitemap:

User-agent: *
Disallow: /admin
Sitemap: https://yourdomain.com/sitemap.xml

This one blocks the whole site, which is a common mistake left over from development:

User-agent: *
Disallow: /

How to check or improve it

1. Open your robots.txt and look for Disallow: / under User-agent: *.
2. Make sure you aren’t blocking search crawlers like Googlebot and Bingbot, or AI search crawlers, by accident.
3. Remember that a crawler follows only the group of rules that names it most specifically, so rules for a named bot replace the general * rules for that bot.
4. Our free audit checks your robots.txt for common problems.