Robots.txt tester
Paste your robots.txt, enter a URL and choose a crawler. See right away if the page is allowed or blocked, and which rule decided it.
What is a robots.txt file?
A robots.txt file sits at the root of your site and tells crawlers which paths they may fetch. It is the first file most search engine bots request, so one wrong line can keep a whole section of your site out of search.
Robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it, just without a description. To keep a page out of the index, use a noindex tag and leave the page crawlable.
How this robots.txt checker decides
The tester follows the rules Google documents. First it picks the user agent group that matches the crawler most specifically, and falls back to the * group if there is none. Within that group the longest matching rule wins, and an Allow rule beats a Disallow rule of the same length.
The * wildcard matches any run of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks every URL that ends in .pdf. Everything runs in your browser, and nothing you paste is sent to us.
Common robots.txt mistakes
The most damaging one is a leftover Disallow: / from a staging site, which blocks everything. Others are blocking CSS and JavaScript files that pages need to render, and adding a group for one bot and forgetting that it no longer follows the * rules.
Test the URLs that matter most, such as your home page, a category page and a recent post, for both Googlebot and Bingbot each time the file changes.
Robots.txt syntax in one minute
A robots.txt file is made of groups. Each group starts with one or more User-agent lines that name the crawler, followed by Disallow and Allow lines that list paths. A Sitemap line can appear anywhere and applies to every crawler.
Paths are case sensitive and always start with a slash. An empty Disallow line means nothing is blocked. Lines that start with # are comments. Google ignores the Crawl-delay directive, while Bing respects it, so do not rely on it to slow Googlebot down.
Google reads only the first 500 KiB of the file and usually caches it for up to 24 hours, so a fix can take a day to be noticed.
How to test a robots.txt change safely
Before you publish, paste the new version into this robots.txt validator and check a short list of URLs: pages that must stay crawlable and pages that must stay blocked. Run the list for Googlebot, Bingbot and the * group, because a crawler with its own group ignores the general rules completely.
After the file is live, open it in your browser to confirm the server returns it as plain text, then keep an eye on the crawl and indexing reports in Google Search Console and Bing Webmaster Tools for the next few days.
Explore more of Search Console Tool
- Google Indexing Tool Submit URLs to Google, Bing and IndexNow
- Bing Search Console Tool Bing Webmaster data next to Google
- Striking Distance Keywords Find rankings in positions 4 to 15
- Content Decay Detector Catch falling pages and cannibalization
- Google Index Checker See if a page is in Google or Bing
FAQ
Robots.txt tester FAQ
Is this robots.txt tester free?
Yes. It is free, needs no login and works entirely in your browser.
Does it fetch my live robots.txt file?
No. Paste the contents of your file into the box. You can open yoursite.com/robots.txt in a new tab and copy it from there. This also lets you test changes before you publish them.
Does it work the same way as Googlebot?
It follows Google's published matching rules: the most specific user agent group, the longest matching path, and Allow winning a tie. Other crawlers may read unusual files slightly differently.
See what Google and Bing really crawl and index on your site
Connect your search consoles and track indexing for every page.