WordPress.org

1. leaderzzz
   Member
   Posted 11 months ago #

   Hi everyone, first post, first wordpress site.

   I am having an issues with google crawling my site, the googlebot is coming back with the following error

   "Googlebot is blocked from http://www.exclusivelybath.co.uk/"

   Originally I had the default wordpress privacy set to hide the site, now i have changed that to allow all search engines.

   I have also added a robots.txt file into the root of the directory.

   pretty much allowing the spiders to crawl everything.

   Where am i going wrong as when i go to google webmaster tools and ask the spider to crawl my site, it keeps coming up with the same error?

   I think this is what is playing up adsense also as it cant crawl the site for 'text' so its not showing ads..
   
   site: http://www.exclusivelybath.co.uk

   Please any help would be appreciated.

   Kind regards

   lee

2. Samuel B
   moderator
   Posted 11 months ago #

   here is your robots.txt:
   

   User-agent: *
   
   Disallow: /feed/
   Disallow: /trackback/
   Disallow: /wp-admin/
   Disallow: /wp-content/
   Disallow: /wp-includes/
   Disallow: /xmlrpc.php
   Disallow: /wp-
   
   Sitemap: http://www.exclusivelybath.co.uk/sitemap.xml

   you're not allowing anything
   look at this example
   http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html

3. leaderzzz
   Member
   Posted 11 months ago #

   Thanks for the reply samuel I have changed it to this:

   User-agent: *
   Disallow:
   Disallow: /cgi-bin/
   Sitemap: http://www.exclusivelybath.co.uk/sitemap.html

   so pretty much any spider can have a party on my website..

   I have just gone back into webmaster tools and google still won't have it!

   -----------------------------------------------------
   This is how Googlebot fetched the page.

   URL: http://www.exclusivelybath.co.uk/

   Date: Sunday, June 12, 2011 1:53:09 AM PDT

   Googlebot Type: Web

   The page could not be crawled because it is blocked by robots.txt. You can find more information in the Help Center article about robots.txt.

   dohh...

   Is there some code in the template stopping google?

   Regards

   lee

4. Samuel B
   moderator
   Posted 11 months ago #

   try this
   http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449&from=40360&rd=1

5. leaderzzz
   Member
   Posted 11 months ago #

   Sam, many thanks for Looking.

   I had tried a compendium of different changes to make it work and when I eventually looked at google's crawl, it had accepted it..

   I did contact the host to see if there was anything related to them.
   I think it was working thereafter, so they may of well had it blocked for some reason..

   Again Thanks for taking the time over the weekend to look.

   Lee

Reply

You must log in to post.