WordPress.org

Ready to get started?Download WordPress

Forums

Robot.txt Blocking Search Engines (7 posts)

  1. David
    Member
    Posted 1 year ago #

    Okay, I'm having a bit of trouble launching a site of mine. Last Friday, I turned off the setting that discourages search engines from index my site. I cleared my cache. I also reinstalled SEO by Yoast. I also submitted my sitemap to Google Webmaster Tools. I know that things don't happen over night, but it's been nearly a week and I still get this error on Google Webmaster Tools. "URL restricted by robots.txt"

    I've looked everywhere on the site and in the source code and cannot figure out why robot.txt are still blocking my site.

    Does anyone have any suggestions on where I am missing information?

    http://www.fenceanddecksupply.com

  2. arncus
    Member
    Posted 1 year ago #

    Hello FenceUtah,

    I've got curious and looked at your site. The main issue is that Google is stating this - so you will need to take a close link at what they're stating in the search results. There's an option there to click (at the end of the search result where it says:

    http://www.fenceanddecksupply.com/
    A description for this result is not available because of this site's robots.txt – learn more.

    If you click on the ROBOTS.TXT link, you'll see what's probably causing the issue.

    Try going into the ROBOTS.TXT file and remove what Google is showing:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Additionally, if you haven't already created an account with Google's Webmaster tools, please use them. They have a robots.txt checker and will probably give you more succinct information.

    [ Signature moderated. ]

  3. David
    Member
    Posted 1 year ago #

    Okay, So I've edited the robots.txt file. I've resubmitted things to Webmaster Tools and cleared the cache.

    I'm still getting the error that it's being blocked.

    Is it just going to take some time?

  4. Krishna
    Volunteer Moderator
    Posted 1 year ago #

    Currently your robots.txt contains:

    User-agent: *
    /

    An entry that applies to all bots looks like this:

    User-agent: *

    So, it is advisable to use only the above entry. Review:
    http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

  5. David
    Member
    Posted 1 year ago #

    Krishna,

    So, if I understand what you're saying, by not stating

    User-agent: *
    /

    that I'm blocking all the robots?

    I've changed it to

    User-agent: *
    Disallow:

    based on the recommendation of http://tools.seobook.com/robots-txt/.

  6. arncus
    Member
    Posted 1 year ago #

    Hello FenceUtah,

    Checkout http://www.robotstxt.org/robotstxt.html and you'll see all of the options that you can use to block or not block bots.

    Krishna is right though - you're blocking everything with your previous entry.

    Arnel C.

  7. David
    Member
    Posted 1 year ago #

    Thanks! I think it's fixed now... we'll see though.

Topic Closed

This topic has been closed to new replies.

About this Topic