WordPress.org

1. didden
   Member
   Posted 11 months ago #
   

   I have a few sites i built this weekend and when I started I disabled the button in WP so would not be crawled yet. I have now enabled that and cannot get the sitemap to go in Webmaster Tools and keeps telling me it is blocked. Three sites one worked just fine and 2 others keep saying blocked.

   If I check my Robots.Txt it is this and just recently changed to this to see if would help.
   User-agent: *

   Disallow: /feed/
   Disallow: /trackback/
   Disallow: /wp-admin/
   Disallow: /wp-content/
   Disallow: /wp-includes/
   Disallow: /xmlrpc.php
   Disallow: /wp-

   Sitemap: http://example.com/sitemap.xml

   In Webmaster tools it says this in one place..
   Home Page Access
   Googlebot is blocked from http://bestgolfingvideos.com/

   This is the error I keep getting.
   URL restricted by robots.txt

   Only 2 plug-ins: Contact us Form and WordPress Seo

   Could it be that since I just hit the button for the robots to start could it take a bit to allow this and just wait.

   Or any suggestions as what to do as to make the 2 sites load the site map in Webmaster like the other 1 did.

   Thanks
   Demnis

2. brainfreeze0
   Member
   Posted 11 months ago #
   

   Hell, even if you go out and flag Google to come crawl your site when you first put it up it can take up to 6 months. Having told them not to crawl it makes it longer.

   Write blog entries and post links on social media and they will crawl through those faster than just coming to visit you on their own.

3. didden
   Member
   Posted 11 months ago #
   

   I understand that but I would also like the site to be crawled and don't understand what is wrong that it is being blocked. So if any ideas as to what to check that might be blocking it I would appreciate it.

   Thanks
   Dennia

Reply

You must log in to post.