Robots.txt checker
The robots.txt file is a short text file at the root of a website that tells search engines and other crawlers which parts of the site they may visit. A single wrong line can hide your whole site from Google.
This tool fetches the file, shows it, breaks it into rules for each crawler, and warns about common mistakes.
How to read the results
- User-agent names the crawler a group of rules applies to. An asterisk means all crawlers.
- Disallow lists paths the crawler should not visit.
Disallow: /for all crawlers blocks everything. - Allow makes an exception inside a disallowed area.
- Sitemap lines point crawlers to your list of pages.
Questions
Does robots.txt hide a page from the public?
No. It only asks well-behaved crawlers to stay away, and anyone can read the file. Use passwords for anything private.
My site is missing from Google. Could robots.txt be the cause?
It is one of the first things to check. Look for a Disallow: / line under User-agent: *.
Is a missing robots.txt a problem?
No. Crawlers will assume everything may be visited. It is still useful to add one so you can list your sitemap.
Related guides
More free tools
- DNS lookup
- HTTP header checker
- SSL certificate checker
- Redirect checker
- WHOIS and domain expiry
- MX record lookup
- Security headers checker
Results are a snapshot taken from our servers at the moment you ask, and can differ from what you see elsewhere. siteupdown.com tests public websites only. Spotted a problem?