WordPress.org

Ready to get started?Download WordPress

Forums

WP Site Blocked from bots (3 posts)

  1. didden
    Member
    Posted 2 years ago #

    I have a few sites i built this weekend and when I started I disabled the button in WP so would not be crawled yet. I have now enabled that and cannot get the sitemap to go in Webmaster Tools and keeps telling me it is blocked. Three sites one worked just fine and 2 others keep saying blocked.

    If I check my Robots.Txt it is this and just recently changed to this to see if would help.
    User-agent: *

    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /xmlrpc.php
    Disallow: /wp-

    Sitemap: http://example.com/sitemap.xml

    In Webmaster tools it says this in one place..
    Home Page Access
    Googlebot is blocked from http://bestgolfingvideos.com/

    This is the error I keep getting.
    URL restricted by robots.txt

    Only 2 plug-ins: Contact us Form and WordPress Seo

    Could it be that since I just hit the button for the robots to start could it take a bit to allow this and just wait.

    Or any suggestions as what to do as to make the 2 sites load the site map in Webmaster like the other 1 did.

    Thanks
    Demnis

  2. brainfreeze0
    Member
    Posted 2 years ago #

    Hell, even if you go out and flag Google to come crawl your site when you first put it up it can take up to 6 months. Having told them not to crawl it makes it longer.

    Write blog entries and post links on social media and they will crawl through those faster than just coming to visit you on their own.

  3. didden
    Member
    Posted 2 years ago #

    I understand that but I would also like the site to be crawled and don't understand what is wrong that it is being blocked. So if any ideas as to what to check that might be blocking it I would appreciate it.

    Thanks
    Dennia

Topic Closed

This topic has been closed to new replies.

About this Topic