Support » Plugin: Yoast SEO » After installing SEO plugin, Google can't access Robots.txt

  • My site has been ranking quite well in the Google search results. However, after I installed SEO plugin, now Google gives me a report saying it can’t access robots.txt file.
    I don’t see a robots.txt file in the directory. Is there a setting in the plugin I need to tweak so I am back on Google results?
    Cheers
    Vedat

    http://wordpress.org/plugins/wordpress-seo/

Viewing 11 replies - 1 through 11 (of 11 total)
  • Thread Starter vedatk

    (@vedatk)

    I deactivated the plugin, created a Robots.txt file and everything went back to normal.
    I then re-activated the plugin thinking the robots.txt file should be fine now. But again, my site started not appearing in search results. There is something that the plugin is doing for Google to ignore the site.
    Unfortunately this plugin is doing exactly the opposite of what I was hoping.
    I need some help to get it up and running.

    I have the same problem, Googlebot can’t access your site
    I have this message in webmaster tools:
    Your site’s overall robots.txt error rate is 100.0%.

    Also in the Titles&Metas -> Post Types, I have selected NoIndex for a few post types, and yet still when I open robots.txt I only have this:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    same problem here, found some info about the WordPress SEO plugin in combination with W3Cash en Cloudflare and try them all but no succes how can help?

    BVRamanarao I don’t have a problem with the sitemap,
    as I stated above my problem is that Googlebot can’t access your site .
    And that the settings I chose in the WordPress SEO plugin for some post types not to be indexed, don’t make any changes to the robots.txt file.

    I can’t find a solution for this, can anyone help?

    Hai igotmore!

    I have the same problem, Googlebot can’t access your site
    I have this message in webmaster tools:
    ‘Your site’s overall robots.txt error rate is 100.0%.’

    and

    I stated above my problem is that Googlebot can’t access your site .

    After reading your problem, I understood that you did not cached what you have to do at your end after reading my previous reply. In my previous reply I have already provided a link where you can find what you must mention in robots.txt (in step 2 of that link, I mentioned that).

    Since most of the sitemap related errors with googlebot when someone opted for wordpress seo sitemaps will be resolved in that post.

    Ok, I explain what you missed it.
    Googlebot search for default sitemaps link that will look like yourdomain.tld/sitemap.xml
    But if we create sitemaps with WordPress SEO plugin then sitemaps index file URI will be yourdomain.tld/sitemap_index.xml

    Googlebot searching for sitemap.xml but not where your site having sitemap_index.xml

    So webmaster must tell the googlebot that his/her sitemap available at sitemap_index.xml, but not at default link ends with just sitemap.xml

    Since Googlebot follows robots.txt directives, you have to mention the following link in your robots.txt

    http://www.YourDomainName.tld/sitemap_index.xml
    or
    http://YourDomainName.tld/sitemap_index.xml

    NOTE: writing www is all depends on your domain URI.

    If you click a sitemap button in WordPress SEO dash board you will get exact URI of your site’s sitemap

    After writing sitemap URI on robots.txt, goto google webmaster tools
    There you have to submit your sitemap and verify, for immediate indexing. Submission to the google webmaster tools is not necessary. Since after reading robots.txt instructions, google find your sitemap as usual, but it takes some time to crawl your site.

    There after you don’t get the error

    Googlebot can’t access your site
    ‘Your site’s overall robots.txt error rate is 100.0%.’

    If still having the same problem first read my previous reply where a link for step by step procedure. Or either provide your sitemap URI, so that we can identify exact issue.

    But I am sure that you must mention your sitemap URI in robots.txt to avoid this error.

    Ok the problem that Googlebot can’t access the site is fixed.

    But I have one question, as I stated my robots.txt looks like this:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    And in the Yoast SEO -> Titles & Metas -> Post Types for the Portfolio and some other post types I have selected noindex, follow

    So shouldn’t that do some changes to my robots.txt to tell the engines not to index that site, my robots.txt hasn’t changed at all after making the changes stated above.

    Yoast SEO offer several options for Metas those can be used for SEO and sitemaps. If you check the noindex, follow then googlebot will not index that specific post or page or other taxonomies. But it will follow the link and Yoast SEO doesn’t restrict googlebot to crawl the said taxonomy. In simple, googlebot crawl your page/post/taxonomy and follow the same even after you check the option noindex, follow. It means Yoast SEO just making your site Most SEO friendly. There are four types of combinations we can get as follows:
    1.index, follow
    [content will be indexed, and followed by bots]
    2.noindex, nofollow
    [content is purely private or duplicate, for this there is an option in WP under settings>>reading>>Discourage search engines from indexing this site]
    3.index, nofollow
    [Suppose you have more than one site or site with more than one sub-domain sharing same content like author page. Googlebot might treat this is a duplicate content. To avoid this one should go with this index, nofollow. But actually this option not provided buy Yoast SEO plugin]
    4.noindex, follow
    [Suppose you don’t want to index particular content or page or post which might not be available after some time. After the due date, if it was indexed, will result 404 error. To avoid that one should go with noindex, follow.
    Another example is that when site is under development and webmaster is not sure about the set of categories and tags that is likely to create or alter. Not indexing those taxonomies until webmaster thinks the site development is over, gives you control over SEO of your site]

    Now come to robots.txt
    There is no reason that one can put their backend directories and sensitive files in their sitemaps. Say, do you prefer to index wp-content directory visible in both sitemaps and search results in google? I hope no one do that. But, bots (including googlebot) crawl your backend directories, cache and even not-found links and make them available to public when some one search in google search engine. But these cann’t be controlled with Yoast SEO.

    Legitimate bots will follow robots.txt instructions, but there are bad/rogue bots and hackers they simply don’t consider these instructions. They crawl as they wish for their ill-full reasons. For legitimate bots giving instructions at robots.txt we can restrict the visibility of backend directories (like wp-content, wp-includes, wp-admin), cache etc in search engines. These features that one cann’t obtain by Yoast SEO. robots.txt just showing the way for bots what to be crawled, indexed and follow.

    To protect from rogue/bad bots and hackers for not to crawl your site’s sensitive areas, you must mention that with rewrite rules in .htaccess. .htaccess gives you an authority, control and security for all the issues discussed above.

    The name of the WordPress SEO itself says that its primary goal is to make your WP site most SEO friendly. robots.txt deals with bots, but you can get greater control with .htaccess. Just for indexing, bypassing, allowing, disallowing things, robots.txt is sufficient enough.
    NOTE: .htaccss in not for beginners.

    Hi,

    I have exactly the same problem.

    I followed the procedure, but google still can’t access my robots.txt file

    I have installed yoast, and everything was fine, i think the problem came when created the sitemap file.

    I updated the robots.txt file to add the link to the sitemap. But since google can’t access that file it doesn’t change anything.

    When i submit the sitemap to google, it says everything is fine. But still an error message for the robots file.

    I have updated the robots file, but when i go on the tool for testing the robots file, the default content is still the old content. I look as if google can’t the see update of the robot file.

    this is the real content of my robots.txt

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/uploads/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/themes/
    Disallow: /wp-content/cache/

    Sitemap: http://www.technologiesdinterieur.com/sitemap_index.xml

    and this is the default content on google tools

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    So it looks the last explored version of the robot is the 6 may. and since then my changes on the robot can’t be read.

    please help, i become crazy

    http://www.technologiesdinterieur.com
    http://www.technologiesdinterieur.com/robots.txt
    http://www.technologiesdinterieur.com/sitemap_index.xml

    Well, I tried to enter my robot.txt file page but all I get are the three lines with User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/ .

    To add the address http://www.YourDomainName.tld/sitemap_index.xml
    or
    http://YourDomainName.tld/sitemap_index.xml
    as suggested, it will not allow me. I can read the file, but cannot enter either of those two address at all.

    I’m going crazy too! Googlebot says it cannot crawl my website anymore…so if wordpress won’t let me alter my file, what can I do??

    Very confusing, how do I modify the robots.txt using Yoast WordRess SEO plugin?

Viewing 11 replies - 1 through 11 (of 11 total)
  • The topic ‘After installing SEO plugin, Google can't access Robots.txt’ is closed to new replies.