WordPress.org

Support

Support » How-To and Troubleshooting » WP Site Blocked from bots

WP Site Blocked from bots

  • I have a few sites i built this weekend and when I started I disabled the button in WP so would not be crawled yet. I have now enabled that and cannot get the sitemap to go in Webmaster Tools and keeps telling me it is blocked. Three sites one worked just fine and 2 others keep saying blocked.

    If I check my Robots.Txt it is this and just recently changed to this to see if would help.
    User-agent: *

    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /xmlrpc.php
    Disallow: /wp-

    Sitemap: http://example.com/sitemap.xml

    In Webmaster tools it says this in one place..
    Home Page Access
    Googlebot is blocked from http://bestgolfingvideos.com/

    This is the error I keep getting.
    URL restricted by robots.txt

    Only 2 plug-ins: Contact us Form and WordPress Seo

    Could it be that since I just hit the button for the robots to start could it take a bit to allow this and just wait.

    Or any suggestions as what to do as to make the 2 sites load the site map in Webmaster like the other 1 did.

    Thanks
    Demnis

Viewing 2 replies - 1 through 2 (of 2 total)
  • Hell, even if you go out and flag Google to come crawl your site when you first put it up it can take up to 6 months. Having told them not to crawl it makes it longer.

    Write blog entries and post links on social media and they will crawl through those faster than just coming to visit you on their own.

    I understand that but I would also like the site to be crawled and don’t understand what is wrong that it is being blocked. So if any ideas as to what to check that might be blocking it I would appreciate it.

    Thanks
    Dennia

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘WP Site Blocked from bots’ is closed to new replies.
Skip to toolbar