The SEO Bots That ~140 Million Websites Block the Most

We looked at the total number of websites blocking the bots. There are many ways to block bots with robots.txt, and this accounts for all of them including:

• Explicit blocks, where the bot is mentioned and disallowed
• General blocks, where all bots may be blocked
• Any instances where a directive allowed the bot, after blocking all bots

Caveats: this doesn’t include any other block types such as firewalls or IP blocks. It is also Ahrefs view of the web, which means if we’re blocked by one of these methods, we may not see the robots.txt files at all.

As I mentioned earlier, the most blocked bot is MJ12bot from Majestic. I suspect there are a few reasons for this.

1. They’re a distributed crawler, meaning you can’t look up or block them by IPs, which makes them less trusted and also more likely to be blocked by user agent.
2. They’ve been crawling the web for longer.
3. They have a smaller user base than more popular SEO tools and therefore less leverage to remove any blocks.

Here are the most blocked SEO bots:

And the total websites blocking SEO bots:

Here’s the data:

It gets a little more complicated to analyze. For the above, we looked at the main robots.txt file for a website, but every subdomain can have their own set of instructions. If we look at the ~461M robots.txt in total, then the most blocked SEO bot is SemrushBot at 5.76%. Here are the top 5:

1. SemrushBot: 5.76%
2. Dotbot (Moz): 5.34%
3. MJ12bot (Majestic): 4.96%
4. BLEXBot: 4.88%
5. Ahrefsbot: 4.67%

For this measure, we’re looking only at cases where a particular bot is disallowed. It does not include any overall disallow statements or cases where only certain bots may be allowed. In these cases, website owners went out of their way to specifically block certain bots.

Majestic’s bot is the most targeted, followed by Moz’s bot.

Here are the most blocked SEO bots by explicit mentions:

Here are the number of websites explicitly blocking SEO bots:

Here’s the data:

We looked at the top 1M sites by DR, which aligns to sites with a DR >45. Semrush is the most blocked followed by Majestic and Moz.

Here’s how it breaks down for each individual bot in different categories of websites. The top 3 are:

1. Autos_and_Vehicles: 39%
2. Books_and_Literature: 27%
3. Real_Estate: 17%