Block Bots By A Hosting Company - A Step Too Far!

On my daily reading today I went into WebProWorld and found a post about a hosting company and can it influence your search engine rank. In that post someone stated that this hosting company had decided to block search engine bots due to usage by particular sites. I was shocked.

In my research on this claim I found it was 'Dreamhost'. It seems that 'Dreamhost' has notified some sites of the need to block search engine crawlers but not other site owners. Also in my research I found a forum with postings from a rep from 'Dreamhost':

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
John system stability manager at DreamHost
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Here is some snippets:

"The only cases where we would block googlebot are when the following conditions are met:

-the site in question is causing the server it is on to be unstable

-the site in question is causing erratic or abnormal behavior on the part of google’s crawler

We do not block googlebot on every busy customer site, only when it is demonstrated that it is causing artificial usage (a 10 page site does not require 5000 hits from googlebot to be indexed =) and when the alternative to blocking googlebot is disabling the entire domain. It is disingenuous to suggest that this refers to google simply indexing sites - I actually have been in direct with google engineers to help sort out the specific cases where by their their crawler was not performing as it should be."

His final comment

"I have already removed the blocks and contacted the customer to apologize - I also have trained the tech responsible and conveyed the proper situations in which such steps would be necessary to our entire company at large".

Read more here:
http://www.seopedia.org/internet-marketing-and-seo/dreamhost-asking-clients-to-block-googlebot/

Scary stuff when you think about it.

Jan