Support » Fixing WordPress » Robot.txt Blocking Search Engines

  • Okay, I’m having a bit of trouble launching a site of mine. Last Friday, I turned off the setting that discourages search engines from index my site. I cleared my cache. I also reinstalled SEO by Yoast. I also submitted my sitemap to Google Webmaster Tools. I know that things don’t happen over night, but it’s been nearly a week and I still get this error on Google Webmaster Tools. “URL restricted by robots.txt”

    I’ve looked everywhere on the site and in the source code and cannot figure out why robot.txt are still blocking my site.

    Does anyone have any suggestions on where I am missing information?

    http://www.fenceanddecksupply.com

Viewing 6 replies - 1 through 6 (of 6 total)
  • Hello FenceUtah,

    I’ve got curious and looked at your site. The main issue is that Google is stating this – so you will need to take a close link at what they’re stating in the search results. There’s an option there to click (at the end of the search result where it says:

    http://www.fenceanddecksupply.com/
    A description for this result is not available because of this site’s robots.txt – learn more.

    If you click on the ROBOTS.TXT link, you’ll see what’s probably causing the issue.

    Try going into the ROBOTS.TXT file and remove what Google is showing:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Additionally, if you haven’t already created an account with Google’s Webmaster tools, please use them. They have a robots.txt checker and will probably give you more succinct information.

    [ Signature moderated. ]

    Okay, So I’ve edited the robots.txt file. I’ve resubmitted things to Webmaster Tools and cleared the cache.

    I’m still getting the error that it’s being blocked.

    Is it just going to take some time?

    Krishna

    (@1nexus)

    Member

    Currently your robots.txt contains:

    User-agent: *
    /

    An entry that applies to all bots looks like this:

    User-agent: *

    So, it is advisable to use only the above entry. Review:
    http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

    Krishna,

    So, if I understand what you’re saying, by not stating

    User-agent: *
    /

    that I’m blocking all the robots?

    I’ve changed it to

    User-agent: *
    Disallow:

    based on the recommendation of http://tools.seobook.com/robots-txt/.

    Hello FenceUtah,

    Checkout http://www.robotstxt.org/robotstxt.html and you’ll see all of the options that you can use to block or not block bots.

    Krishna is right though – you’re blocking everything with your previous entry.

    Arnel C.

    Thanks! I think it’s fixed now… we’ll see though.

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘Robot.txt Blocking Search Engines’ is closed to new replies.