robots.txt explained: what you block and what you don't on a business website

robots.txt is a tiny text file that sits at the root of your website and tells search engine crawlers where they're allowed to wander. It sounds technical, but the decision that matters is simple: what you leave visible and what you keep aside. And this is where many business owners make an expensive mistake — they block exactly the pages that should be showing up in Google.

Good news: you don't need to be a developer to understand it. In this article I'll show you, in plain language, what robots.txt is for, what it is NOT for (the part people confuse most often), how to check in two minutes what your own site has, and when it's actually worth touching. No unnecessary jargon.

What robots.txt actually is

Picture a sign at the entrance of a building: "Reception and showroom — welcome; warehouse and accounting office — no visitors." That's roughly what robots.txt does for search engine crawlers (Googlebot and the rest).

It's a plain text file, always available at the same address: your-domain.com/robots.txt. You can open it in your browser right now and see what's written there — any public site shows it to anyone.

The rules inside look like this:

• User-agent: * — "this rule is for all crawlers"
• Disallow: /admin — "don't enter the /admin area"
• Allow: / — "the rest is open"

Serious search engine crawlers read it and respect it. It's a convention of good manners, not a lock — keep that in mind, it matters for the next section.

The big confusion: "blocked" doesn't mean "hidden"

This is where almost everyone goes wrong. People think that if they put a page in Disallow, it disappears from Google. False — and it's a confusion that can do damage.

Disallow tells crawlers "don't read the content of this page." It doesn't tell them "forget it exists." If another site links to that page, Google can still display the address in results, just without a description — it looks messy and out of your control.

So robots.txt is NOT a security method. Don't hide pages with sensitive data, invoices, client areas, or anything you don't want anyone to see. Whoever knows the address walks right in; the file stops no one.

• Want a page truly out of Google? You use a "noindex" meta tag on the page, not robots.txt.
• Want something genuinely private? You protect it with a password or authentication, at the server level.

What's worth blocking and what must stay open

For a normal business website — presentation, services, portfolio, contact — the golden rule is: leave everything open, block only the technical clutter.

Typically worth keeping away from crawlers:

• admin areas (e.g. /wp-admin on WordPress)
• cart, checkout, and customer account pages in a shop
• internal search results and filters that generate thousands of duplicate addresses
• technical files no one searches for anyway

What must NEVER be blocked if you want customers from Google:

• the homepage
• the services and pricing pages
• blog articles
• CSS and JavaScript files — Google needs them to "see" how the site looks; if you block them, it may think the site is broken on mobile

The classic mistake is a single line, "Disallow: /", left over from the build phase. It says "index NOTHING" and pulls your entire site out of search.

How to check your own site in two minutes

You don't need expensive tools. Three steps:

1. Open your-domain.com/robots.txt in your browser. If you see "Disallow: /" on its own, stop — you have a problem that needs fixing urgently.

2. Go to Google and search: site:your-domain.com. It shows you which of your pages Google knows about. If important pages are missing, something is blocking them.

3. Use Google Search Console (free) — it has a robots.txt testing tool and tells you exactly which page is blocked and why.

If you're on a ready-made platform (WordPress, Shopify, wpcom), robots.txt is often generated automatically and correctly from the start. Problems appear when someone edited it manually or left the "discourage search engines" setting ticked from the build phase.

When it's worth calling someone

For most business sites, a correct robots.txt basically means "don't touch it, let the platform generate it." Complexity appears with large shops, sites with many filters, or migrations, where duplicate addresses genuinely confuse Google.

At MPO Web Studio, when we build a site we deliver it with robots.txt and indexing thought through from the start — sales pages open, technical areas closed, without the fatal line that hides everything. We work entirely remotely, across the whole country, and we can first build you a ready-made demo of your site so you see the result before you pay anything.

If you suspect your site "isn't showing up on Google" and don't know where to start, message us on WhatsApp. We'll look at robots.txt and indexing together and tell you honestly whether it's a five-minute fix or something more serious — with no "guaranteed #1 spot" promises, because no honest person can guarantee that.