Robots.txt is driving me insane…anyone

leaderzzz
(@leaderzzz)

12 years, 10 months ago

Hi everyone, first post, first wordpress site.

I am having an issues with google crawling my site, the googlebot is coming back with the following error

“Googlebot is blocked from http://www.exclusivelybath.co.uk/”

Originally I had the default wordpress privacy set to hide the site, now i have changed that to allow all search engines.

I have also added a robots.txt file into the root of the directory.

pretty much allowing the spiders to crawl everything.

Where am i going wrong as when i go to google webmaster tools and ask the spider to crawl my site, it keeps coming up with the same error?

I think this is what is playing up adsense also as it cant crawl the site for ‘text’ so its not showing ads..

site: http://www.exclusivelybath.co.uk

Please any help would be appreciated.

Kind regards

lee

Viewing 4 replies - 1 through 4 (of 4 total)

Samuel B
(@samboll)

12 years, 10 months ago
here is your robots.txt:
```
User-agent: *

Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-

Sitemap: http://www.exclusivelybath.co.uk/sitemap.xml
```
you’re not allowing anything
look at this example
http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html
Thread Starter leaderzzz
(@leaderzzz)

12 years, 10 months ago

Thanks for the reply samuel I have changed it to this:

User-agent: *
Disallow:
Disallow: /cgi-bin/
Sitemap: http://www.exclusivelybath.co.uk/sitemap.html

so pretty much any spider can have a party on my website..

I have just gone back into webmaster tools and google still won’t have it!

—————————————————–
This is how Googlebot fetched the page.

URL: http://www.exclusivelybath.co.uk/

Date: Sunday, June 12, 2011 1:53:09 AM PDT

Googlebot Type: Web

The page could not be crawled because it is blocked by robots.txt. You can find more information in the Help Center article about robots.txt.

dohh…

Is there some code in the template stopping google?

Regards

lee

Samuel B
(@samboll)

12 years, 10 months ago

try this
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449&from=40360&rd=1

Thread Starter leaderzzz
(@leaderzzz)

12 years, 10 months ago

Sam, many thanks for Looking.

I had tried a compendium of different changes to make it work and when I eventually looked at google’s crawl, it had accepted it..

I did contact the host to see if there was anything related to them.
I think it was working thereafter, so they may of well had it blocked for some reason..

Again Thanks for taking the time over the weekend to look.

Lee

Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘Robots.txt is driving me insane…anyone’ is closed to new replies.

Robots.txt is driving me insane…anyone

Tags

Topics

Topics with no replies

Non-support topics

Resolved topics

Unresolved topics

All topics