What does 'Disallow: /' mean?

Every crawler is told to stay out of the whole site. Nothing on the site breaks, which is why it goes unnoticed — usually a leftover from a staging or development copy — but Google cannot read any page while it stays. A blocked page can still be listed from links elsewhere, without its content or a proper description.

My site is not in Google. Is this why?

It is the first thing to check. If robots.txt blocks the site, or the sitemap is missing or empty, Google has nothing to work with. After fixing it, submit the sitemap in Search Console and request indexing for the homepage.

For a five-page site Google will find everything by following links. Beyond that, a sitemap is how new and changed pages get found quickly, and lastmod dates tell Google what to re-crawl. Every WordPress SEO plugin makes one.

Why does it check whether CSS and JavaScript are blocked?

Google renders pages like a browser. If robots.txt blocks the styles or scripts folder, Google may render the page without its layout, and judge it on a broken version of what visitors see.

Can this check a site behind Cloudflare?

Usually. robots.txt and sitemap.xml are plain files served to any client, and bot protection set up for pages is often not applied to them. If the host blocks this check too, the result says the file could not be read.

Robots & Sitemap Checker

A light, automated check — not the deep audit. Enter your site; the server reads /robots.txt and the XML sitemap it names (or /sitemap.xml) and reports whether search engines are allowed in, whether styles and images are blocked, and whether the sitemap exists, is valid and lists your pages.

A free robots.txt and sitemap checker: enter your site and see whether search engines are allowed to crawl it, whether styles and images are blocked, whether a sitemap is listed and valid, and how many URLs it holds — with a fix for each finding. It reads the same two files Google's crawler reads; nothing is stored.

One line in robots.txt can stop Google crawling your whole site — and nothing on the site looks broken.

Your robots.txt and sitemap, read the way Google's crawler reads them.

The page you enter is fetched once by our server (public sites only), read, and discarded — nothing is stored, and your own files never come into it.

After the check

Your site, looked after every week

Left: your site as it stands today, with whatever this check finds and the things it cannot see from outside. Right: the same site with my team looking after it. It is checked every week, a backup is taken before anything changes, and if something breaks, my team looks into it and tells you what happened.

The findings fixed, in the order they matter
Checked again every week, with a backup taken first
Someone watching: uptime, speed, what Google sees

900+ projects
since 2017: 4.9★
400+ five-star reviews on Fiverr: 9+ years
building WordPress and Next.js sites

Support is the whole team, not one person — reach us any time, and someone picks it up.

“Working with Sandipan Das was fantastic! The attention to detail and professionalism in his work was outstanding, and his proactive communication and timely delivery made the entire process smooth. He crafted the website exactly as I envisioned—highly recommend! 👍”

Indoor Pro Golf · 3-page WordPress site · the project

Schedule a Meeting

Bring what the tool found. I lead every project, and on the call we go through the list together and decide what to fix first.

Schedule a Meeting

How it works

How to use the Robots & Sitemap Checker

01Enter the site address — the homepage is enough.
02Read the findings: whole site blocked, assets blocked, sitemap listed, sitemap valid, URL count, lastmod dates.
03Fix what is red — usually one line in robots.txt or one plugin setting — then resubmit the sitemap in Search Console.

Good to know

The mistake this catches most

A site launched with the staging site's robots.txt still in place: 'Disallow: /'. The site looks fine and nothing is broken, but Google is told not to crawl any of it. This check reads the robots.txt file itself and the rules written for Googlebot (or for every crawler), so a homepage that opens normally does not hide the problem.

In more depth

About the robots & sitemap checker

The two files Google reads before anything else

Before a crawler looks at a single page it fetches /robots.txt to learn what it may visit, and it looks for an XML sitemap to learn what exists. A mistake in either one is invisible to visitors and fatal for search: the site works, the analytics look normal, and nothing new ever ranks.

This checker reads both files, applies the rules written for Googlebot (or for every crawler), and puts the findings in plain language: is the site blocked, are assets blocked, is the sitemap listed, valid and populated, when was it last updated. Red findings are usually one line to fix.

FAQ