Robots.txt generator and tester

Download robots.txt and compare it with any rules you already have. Serve it as UTF-8 plain text at the root of the site, such as https://example.com/robots.txt, with a successful HTTP response. A file at /blog/robots.txt does not apply to /blog/. Upload it with your host's file manager or your deployment process, then open the address to see what is really served. This tool does not upload or install anything.

A file applies only to the scheme, host and port that serve it. HTTPS and HTTP, example.com and www.example.com, other subdomains and non-standard ports can each need their own robots.txt. Paths in the rules are relative to that site, and a sitemap can be hosted elsewhere. The tester does not check which site a file belongs to.

Use crawler names such as Googlebot or Bingbot, one per line, not browser User-Agent text. Names are matched without regard to case. A crawler follows only the group that names it. Groups that name the same crawler are combined, and the group for * is used only when no group names the crawler. Googlebot-Image, Googlebot-News, Googlebot-Video, Google-InspectionTool and Google-CloudVertexBot follow the Googlebot group, and GoogleOther-Image and GoogleOther-Video follow the GoogleOther group, when they have no group of their own. Google lists each of these crawlers with a second name for this reason. Crawler names that differ only in case are written once. Paths are case sensitive. The longest matching rule wins, and Allow wins a tie. A rule with no path does nothing.

A rule starts with / and matches from the start of the path. * matches any run of characters, including none. A final $ matches the end of the address, query string included, so /*.php$ matches /page.php but not /page.php?id=1. Write % escapes for spaces and special characters: %23 for #, %2A for * and %24 for $. Text outside ASCII is compared as UTF-8, and an escaped plain character such as %7E means the same as ~, but %2F is not the same as /. The tester reads a path the way a browser does, so /a/../b is tested as /b and characters that browsers always escape, such as { } < > " and `, are escaped. Write those characters in a rule as %7B, %7D, %3C, %3E, %22 and %60, or the rule will not match. Use the tester on the exact paths you care about.

Public website puts no limits on crawling. WordPress blocks /wp-admin/ and allows /wp-admin/admin-ajax.php, which assumes WordPress is installed in the site root; change the paths if it is in a subfolder, and check your plugins. WordPress may already serve a virtual robots.txt, and a real file in the root replaces it, so keep the rules and sitemap addresses you need. Block crawling adds Disallow: / for all crawlers, and groups you add for named crawlers override it for those crawlers. No preset makes up a sitemap, a list of private paths or a promise about search engines.

No. A search engine can still list a blocked address that it finds through links, without any of the page's content. To keep a page out of results, use a noindex meta tag or an X-Robots-Tag header on a page that crawlers are allowed to fetch. Blocking the page in robots.txt stops crawlers from seeing the noindex. For private material, require a login.

Yes, keep the files that search engines need to render your pages. Broad rules for /wp-content/, /assets/ or file extensions can hide them. Check which files your pages load before you add such a rule. The WordPress preset blocks only the admin area.

No. The file is public, and well-behaved crawlers choose whether to follow it. A path listed in it can even show where something is. Protect private files with a login and access rules on the server, and handle abusive traffic on the server too. Do not put passwords, tokens or private addresses in comments or sitemap lines.

The tester checks only the groups and rules on this page. It does not fetch your site, follow redirects, repeat how a server error is handled, or look at a crawler's cache. Crawlers can keep an old copy of robots.txt, differ in small ways, or ignore the file. After you publish, open the file and use the crawler provider's own tools to check it.

The file uses User-agent, Allow, Disallow, Sitemap and comment lines. Crawl-delay, Noindex, Host and other lines are not written; Google ignores crawl-delay. The limits are 20 groups, 20 crawler names per group, 500 rules per group, 2,000 rules in all, 50 sitemap addresses, 3,000 lines and 500 KiB of UTF-8 text, which is the most Google reads. A rule path can be up to 2,048 characters. The tester stops without an answer when wildcard rules are too complex to check. While anything is wrong with your settings, Copy and Download stay off.

No. The file is made, tested, copied and downloaded in this browser, and the test addresses are never requested. Nothing is stored. Reset clears every field and goes back to the public website preset.

Read RFC 9309: Robots Exclusion Protocol, Google's robots.txt specification and Google's guide to creating a robots.txt file. Sitemap lines are a widely supported addition, not a rule about access.