SEO
August 7, 2024
Robots.txt is a file that helps search engines find what pages on your website aren’t accessible to the bots.
Whenever a bot enters your website it needs to go through the gates wherein it is told which pages the crawlers should not access.
This is especially important to hide pages that you don’t want to show publicly to search engines (like login pages, your internal data if on your web pages)
Most websites don’t need a robots.txt file.
Over the years the Google algorithms have become pretty smart. It can usually find and index all the important stuff on your site without any help.
But there are a few reasons why you might want to use one:
But here’s the thing – it’s not foolproof. Good bots (like Google) will follow the rules, but bad bots (like spam bots) might ignore them completely.
So, is robots.txt important?
It can be, especially if you’ve got a big site or stuff you want to keep private. But for most small websites, it’s not something you need to lose sleep over.
You can check all of your indexed pages in your search console from the pages section.
If there is a mismatch in the number of pages you want to get indexed, you need to check your robots.txt.
Alright, let’s break this down in simple terms. These three things robots.txt, meta robots, and X-robots – all tell search engines what to do with your website’s content. But they’re not the same thing. Here’s the deal:
Robots.txt: It’s a file that sits right at the website’s root directory. Its job is to tell search engine crawlers which parts of your site they can and can’t look at. It’s like giving them a map of your website and saying, “You can go here, but not there.”
Meta robots tags: These are more like little signs on each page of your website. They’re bits of code that you put in the <head> section of every webpage. Each sign tells search engines two things about that specific page:
Look at the image below, this code is at the page level.
X-robot tags: These are special signs for stuff that isn’t regular web pages, like PDFs or images. You put these tags in something called the HTTP header of the file. It’s a way to give instructions about files that don’t have a <head> section like regular web pages do.
Robots.txt is for your whole site, meta robots are for individual pages, and X-robots are for special files. Each has its place in telling search engines what’s on your site.
To use robots.txt, you’ve got to create one first. It’s super simple:
User-agent: X
Disallow: Y
What does this mean?
Let’s say you don’t want Google looking at your images. You’d write:
User-agent: googlebot
Disallow: /images
Want to talk to all bots at once? Use an asterisk (*) like this:
User-agent: *
Disallow: /images
This tells all bots to stay out of your images folder.
Remember, this is just scratching the surface. There’s a lot more you can do with robots.txt. Google’s got a guide that you can look into to understand all the rules you can use.
The main thing is to keep it simple. Start with the basics, and you can always add more rules as you need them.
Alright, you’ve made your robots.txt file. Now let’s get it up and running.
You can put your robots.txt file pretty much anywhere on your site. But if you want to make sure bots find it easily, stick it here:
https://example.com/robots.txt
Quick heads up: The name of the file matters. Make sure the “r” in “robots.txt” is lowercase.
You’ve got to get your robots.txt file right. One little mistake could mean your whole site disappears from search results. Yeah, it’s that serious.
You can test your robots.txt file using this free tool. Use it to make sure your robots.txt is doing what you want it to do.
Remember, it’s always better to double-check. Take a few minutes to run your robots.txt through this tool. It could save you a big headache down the road.
A robots.txt file is like a set of instructions for search engine bots. Here’s how it’s put together:
Let’s look at a simple example:
User-agent: Bingbot
Disallow: /secret-stuff
User-agent: Yandexbot
Disallow: /not-for-yandex
Sitemap: https://www.yourwebsite.com/sitemap.xml
Breaking it down:
1. User-Agent: This is the bot’s name.
For example, if you don’t want Bing to look at your WordPress admin page:
User-agent: Bingbot
Disallow: /wp-admin/
2. Disallow: This tells the bot where it can’t go.
You can have multiple “disallow” lines. If you leave it empty, it means the bot can go anywhere.
3. Allow: This is the opposite of disallow.
It lets you say “yes” to specific pages, even if you’ve said “no” to the whole folder.
For example:
User-agent: Bingbot
Disallow: /blog
Allow: /blog/cool-post
This tells Bing not to look at your blog, except for one specific post.
4. Sitemap: This is like giving search engines a map of your website. It usually goes at the top or bottom of your robots.txt file:
Sitemap: https://www.yourwebsite.com/sitemap.xml
5. Crawl-delay: This tells bots to slow down if they’re putting too much strain on your server.
For example: User-agent: * Crawl-delay: 10
This tells all bots to wait 10 seconds between each page they look at.
Different search engines might handle these instructions slightly differently. It’s always a good idea to check how your specific target search engines interpret robots.txt files.
Best Practices of using robots.txt:
Limitations of robots.txt:
The examples below have all these syntax used, you will better get the grasp of each, also we have explained why each robots.txt below is different from each other and what purpose each have.
In this section, we will try to understand the robots.txt files of some of the giants in the search industry. I will let you go through some parts (or all) of the files to make you better understand each.
Let’s begin!
Forbes uses a two-tiered approach in their robots.txt file: (this is for the screenshot we see above, the original robots.txt file is much larger and it was hard to include the whole file in this blog)
For all user agents (*):
For Googlebot news specifically:
This strategy allows Forbes to:
Capterra’s robots.txt is more specific:
This approach helps Capterra:
SEJ’s file is highly technical:
This configuration allows SEJ to:
Trustpilot focuses on protecting user data and managing their review content:
This helps Trustpilot:
TripAdvisor has an extensive robots.txt file:
This comprehensive approach allows TripAdvisor to:
Each of these robots.txt files reflects the complex needs of large, content-rich websites, balancing SEO needs with user privacy and content management.
Using robots.txt allows you to have a better control over the indexability of your website. It is advisable for every legitimate business to use one.
However, having a good robots.txt file in your website’s root directory does not guarantee SEO success. Robots.txt is just another part of the whole SEO campaign.
Just a little about us, we are Serpple, an SEO tool, which can help you to do keyword research, track your google ranking position and manage your backlinks.
We have been building an SEO tool for businesses that need action based data. Whether you’re just starting with SEO or looking to refine your strategy, tools Serpple’s tools can help you make the most of your efforts,
