Futurism logo

Why blocking a page doesn't hide it from Google

SEO Fix Friday, episode three: robots.txt, noindex and canonical sound like the same thing, but they do three different jobs.

By Gautam AtmakuriPublished 2 days ago • 4 min read

Blocking a page in robots.txt is one of the most reliable ways to make sure Google never hears you asking it to remove that page. It sounds backwards. That's still how the plumbing works.

SEO Fix Friday is my weekly series: one technical idea, about a minute of video. Episode one was the map of how search works, and episode two was how a page feels once someone lands on it. This one is about telling Google what to do with pages you don't want shown, or pages that get shown twice.

Three tools get lumped together here: robots.txt, noindex and the canonical tag. They sound like cousins. They're more like three tradies who all turn up in the same kind of ute and do completely different jobs.

robots.txt manages the crawl

A robots.txt file tells crawlers which URLs on your site they're allowed to access. Google's documentation says it's used mainly to avoid overloading your site with requests. Think traffic control.

Picture a "Keep off the grass" sign at the park. It stops people walking on the lawn, but the lawn doesn't vanish, and anyone on the footpath can still tell their mates it's there. A URL disallowed in robots.txt works the same way. It can still be indexed if other sites link to it, and it can turn up in results without a description, because Google never got to read the page.

It isn't a security tool either. If something needs to be private, put it behind a password. A sign on the lawn has never stopped a determined dog.

noindex is the one that removes a page

Of the three, this is the tag that actually controls indexing. You set it with a meta tag on the page, or with an HTTP response header called X-Robots-Tag. When Googlebot crawls the page and sees it, Google drops that page entirely from results, no matter who links to it.

Think of a note on the fridge for your flatmate: "Please take my name off the power bill." It works, but only once they walk into the kitchen and read it. Google has to crawl the page to see the tag, so never block a noindex page in robots.txt. And writing noindex inside robots.txt itself isn't supported by Google, so that shortcut goes nowhere.

Don't expect it to be quick. Depending on how important the page is, Googlebot can take months to come back round. If you're in a hurry, the URL Inspection tool in Search Console lets you request a recrawl.

canonical picks the main version

Then there's the canonical tag, rel="canonical". It tells Google which of several duplicate or very similar pages you prefer. It also pulls signals, such as links, into that one URL.

Say you come home from Mission Bay with six nearly identical shots of the same sunset. You print the best one and stick it on the fridge. The other five still exist on your phone; you've just said which one counts. Google treats that choice as a strong signal, and it can still overrule you. None of these methods are compulsory, and if you don't choose, Google will pick a version itself.

Google's docs add two warnings. Don't use noindex to choose a canonical within your site, because it blocks that page from Search completely. And don't use robots.txt for canonicalisation either.

The classic mistake

Someone wants a page gone, so they add noindex. Then, to be extra sure, they block it in robots.txt as well. Now Googlebot can't fetch the page, so it never reads the noindex, and the page can hang around in results on the strength of links from elsewhere. Belt and braces, except the belt stops anyone reaching the braces.

Mr. Mummu, who wanders into these posts now and then, made the same mistake with a garage sale. Rain came in, so he called it off. He wrote "SALE CANCELLED, GO HOME" in his notebook titled "Urgent", between "onions" and "more onions", tore the page out and taped it inside the garage door. Then, so nobody would wander in and get wet, he hung a second sign outside: "DO NOT OPEN". Two signs, one strip of tape. He was very pleased with the efficiency. By ten o'clock there were eleven people on his driveway, politely not opening the door, waiting for a sale nobody had told them was off. Mummu watched through the window with his chai, then frowned at his notebook. "Ah," he said. "If they can't open the door, they can't read the door."

Where I land on this

My view: most of the mess comes from using robots.txt as a delete button. It's a crawl manager. So ask what you actually need first. Stop the crawl, drop the page, or pick the main version. Then use the matching tool.

One honest limitation. All of this is slower and softer than people expect. A noindex on a page Google rarely visits may sit unread for a long while, and the canonical tag is a hint that Google may quietly disagree with. Everything above is also a simplified version of Google's Search Central documentation, with the edge cases left out.

Every site is different, so treat this as general education, not advice for your specific site.

This week's video runs about a minute. It's episode three of SEO Fix Friday on my YouTube channel, and I'll drop it in the comments.

Your robots.txt decides who gets through the gate. It can't deliver a message to someone it has already turned away.

techsciencehow to

About the Creator

Gautam Atmakuri

I once spent seven days blindfolded, silent and without food to see what my senses were hiding. I write about that, my baby boy, barefoot running, Tirupati to Auckland life, and the business, marketing and tech I build at AumnI digital.

Enjoyed the story? Support the Creator.

Subscribe for free to receive all their stories in your feed. You could also become a paid subscriber, letting them know you appreciate their work.

Subscribe For Free

Reader insights

Comments

There are no comments for this story

Be the first to respond and start the conversation.

Sign in to comment
    Written by Gautam Atmakuri