SEO 4 Videos is a modern digital marketing and SEO firm that will enhance your brand to make one metric explode.

Location

Phone : 651-703-2336

1100 County Road D W,

New Brighton, MN 55112

Opening Time

Tuesday: 10:00am - 07:00pm

Wednesday: 10:00am - 07:00pm

Thursday: 10:00am - 07:00pm

Friday: 10:00am - 07:00pm

Saturday: 10:00am - 07:00pm

Why Googlebot May Be Hitting Bot Blocker URLs

Many sites use bot blockers like Distil Networks to stop scrapers from stealing their data. The challenge for SEOs is that sometimes these bot blockers are not set correctly and can prevent good bots like Googlebot and BingBot from getting to the content, which can cause serious SEO issues. Distil is pretty adamant that their service is SEO-safe, but I am not so certain about others. We recently saw a case where the http version of a big site’s homepage was sending Googlebot to a 404 URL while sending users to the https homepage, all because the bot blocker (not Distil) was not tuned correctly. Was kind of funny searching for their brand and sometimes seeing “Oops” as the first result. It wasn’t that funny for the client.

So when we see odd crawling behavior or other odd signals, we immediately start wondering if it could be the bot blocker.

This week we saw URLs like /distil_r_blocked.html?… start to show up in a client’s GSC crawl errors report under the Other tab. I immediately went into holy bot blocker Batman! mode. You definitely don’t want Googlebot hitting bot blocker URLs.

But after doing a bit of forensic research, I realized that this was actually a case of the bot blocker working properly. Viewing the error details info in GSC showed that the source of the Distil URL was coming from another domain:
Bot-Blocker-Googlebot-SEO Why Googlebot May Be Hitting Bot Blocker URLs

It looks as though these guys tried to scrape the site, hit Distil URLs and then published them as links. So when Googlebot crawled them (why Googlebot would want to crawl these crap sites is another issue) it would hit the Distil URLs.

I am still highly suspicious of bot blockers when it comes to SEO, because of their under-the-hood nature, but next time you see Distil URLs show up in GSC, don’t panic. It may just be those crap links your site is so good at attracting.

The post Why Googlebot May Be Hitting Bot Blocker URLs appeared first on Local SEO Guide.

LocalSeoGuide?d=yIl2AUoC8zA Why Googlebot May Be Hitting Bot Blocker URLs LocalSeoGuide?d=7Q72WNTAKBA Why Googlebot May Be Hitting Bot Blocker URLs LocalSeoGuide?i=4bVLJTN27J0:64ibI59ORhs:V_sGLiPBpWU Why Googlebot May Be Hitting Bot Blocker URLs LocalSeoGuide?d=qj6IDK7rITs Why Googlebot May Be Hitting Bot Blocker URLs LocalSeoGuide?i=4bVLJTN27J0:64ibI59ORhs:KwTdNBX3Jqk Why Googlebot May Be Hitting Bot Blocker URLs LocalSeoGuide?i=4bVLJTN27J0:64ibI59ORhs:gIN9vFwOqvQ Why Googlebot May Be Hitting Bot Blocker URLs LocalSeoGuide?d=TzevzKxY174 Why Googlebot May Be Hitting Bot Blocker URLs LocalSeoGuide?d=l6gmwiTKsz0 Why Googlebot May Be Hitting Bot Blocker URLs

4bVLJTN27J0 Why Googlebot May Be Hitting Bot Blocker URLs