WordPress.org

Ready to get started?Download WordPress

Forums

[resolved] Googlebot is blocked by robots.txt (12 posts)

  1. tashb
    Member
    Posted 1 year ago #

    I have a problem with getting Google to index my WordPress site.

    I am getting errors and warnings saying "Googlebot is blocked" from my site and "URL restricted by robots.txt" in my sitemap uploads.

    Initially while I was creating my website I ticked the 'Ask search engines not to index this site" in the Privacy settings. I have now ticked "Allow search engines to index this site" option, and I have checked http://www.recphysiomassage.com.au/robots.txt. It looks fine and when I test in ismyblogworking.com. it confirms that my robots.txt exists and allows search engines to index.

    Has anyone experienced this problem? Is there a solution I can try to overcome the googlebot problem? Will the problem go away when Google next tries to index my site or will it remain blocked?

    I am very concerned that I will not be able to get my site listed on Google searches because of this.

    Thank you,
    Tash

  2. Pioneer Valley Web Design
    Member
    Posted 1 year ago #

    Check:

    Admin Dashboard>Settings>General>Privacy

    What is setting?

  3. tashb
    Member
    Posted 1 year ago #

    The Privacy setting is currently "Allow search engines to index this site". I changed this yesterday.

  4. Pioneer Valley Web Design
    Member
    Posted 1 year ago #

    I changed this yesterday.

    Use Google Webmaster Tools to re-index and re-submit your robots.txt file. If using a caching plugin make sure to flush it.

  5. tashb
    Member
    Posted 1 year ago #

    Hi Seacoast,

    Thanks for your help. Can you please explain to me what a caching plugin is and also, how would I go about flushing this if I am using one?

    Thank you,
    Tash

  6. tashb
    Member
    Posted 1 year ago #

    Thanks, Seacoast.

    I googled 'caching plugin' and found out you can add a caching plugin to your website that generates static html files from the php code, improving browsing performance for people viewing the site.

    I'm not using a caching plugin so I don't have any html files to flush/regenerate.

    Cheers,
    Tash

  7. tashb
    Member
    Posted 1 year ago #

    Has anyone else experienced the googlebot robots.txt blocking problem when they've changed the site Privacy setting from "Ask search engines not to index this site" to "Allow search engines to index this site"?

    Does this problem go away by itself or will Google not be able to index my site? What can I do to resolve this?

    Thanks for your help,
    Tash

  8. Pioneer Valley Web Design
    Member
    Posted 1 year ago #

    A) Please provide a site link for us to review all this. Without, your queries are moot.

    B) Server caching, when done properly with a supported and well maintained plugin such as WP Super Cache 'records' the generated source code (HTML) in a simple manner and sends that to your site visitors. It reduces the page load time and combined with proper local browser caching rules allows the local browser to carry the weight on retun visits.

    As for Google - wait. Add good and unique content. It's not overnight.

  9. tashb
    Member
    Posted 1 year ago #

    Thanks, Seacoast. Site link is pretty much as above without the robots.txt - http://www.recphysiomassage.com.au

    I'll try WP Super Cache too. Our page load time is very slow so this is very useful to know.

    Will work on improving the content also, and wait :-)

    Cheers,
    Tash

  10. ruatuki
    Member
    Posted 1 year ago #

    I have a similar problem described here
    http://wordpress.org/support/topic/robotstxt-and-google?replies=11 and despite it being set as 'resolved' it is still a problem and I don't have a solution.

  11. Pioneer Valley Web Design
    Member
    Posted 1 year ago #

    @ruatuki, the file does not appear to be web readable - contact your webhost and seek their assistance in resolving this issue.

  12. tashb
    Member
    Posted 1 year ago #

    I waited a couple of days, then went back into Google Webmaster Tools. When I went to Fetch as Google & hit the 'Fetch' button, the status was 'succeeded' and I was able to submit the index.

    I did nothing other than wait, although I did send a tweet with my website address referenced in it in the hope this would make Google crawl my site again sooner, so as to find the new, non-blocking, robots.txt file.

    Just wanted to provide an update that waiting works.

    Thanks to everyone for their help.

Topic Closed

This topic has been closed to new replies.

About this Topic