WordPress.org

Ready to get started?Download WordPress

Forums

WordPress SEO by Yoast
Robots (10 posts)

  1. shnozolla
    Member
    Posted 3 years ago #

    Hi,

    I have just installed the WordPress SEO extension by yoast, I have selected a few of the settings under Indexation rules for the robots.

    I have not seen any change to /robots.txt and I have not seen any robots meta tags added to any of the pages.

    Am I missing something?

    Thank you kindly.

    http://wordpress.org/extend/plugins/wordpress-seo/

  2. illuminea
    Member
    Posted 3 years ago #

    Hi - I have the same question: when sitemaps are activated in WordPress SEO plugin, a path to the sitemap is not added to robots.txt. Do we need to add it manually? If so, is there any recommended practice for doing so?

    Thanks!

  3. illuminea
    Member
    Posted 3 years ago #

    Never mind - I found the answer in your post here:

    http://yoast.com/xml-sitemap-in-the-wordpress-seo-plugin/

    shnozolla, the reason it doesn't appear in robots.txt is that it's apparently not necessary.

    Thanks!

  4. shnozolla
    Member
    Posted 3 years ago #

    Hi Miriam.

    I think we were asking different questions:

    I was asking why no CONTENT has been written in robots.txt.
    How do the search engines know which links to follow and which ones not to follow?

    Thanks.

  5. illuminea
    Member
    Posted 3 years ago #

    shnozolla, I believe that the plugin depends on robots meta info that appears at the top of every page to tell the search engines which content to index and follow. The reason, if I understand correctly, is that robots.txt files full of links is great fodder for spammers and other shady types to find content you don't want them to find.

    Miriam

  6. shnozolla
    Member
    Posted 3 years ago #

    OK, that makes sense.

    Part of my original question was: "...and I have not seen any robots meta tags added to any of the pages."

    Any ideas?

    Thank you Miriam.

  7. illuminea
    Member
    Posted 3 years ago #

    shnozolla, are you sure you're not seeing any robots meta tags? Can you share a link? What version of WP are you running?

    If they are not showing up, could be a number of things (I'm partially guessing here):

    1. Incompatibility with the version of WP
    2. Conflict with some other plugin(s)
    3. Something in your theme that doesn't allow the plugin to do its stuff (I'm not sure if this is possible)

    Miriam

  8. shnozolla
    Member
    Posted 3 years ago #

    Here is the link.
    Version 3.1.1

  9. illuminea
    Member
    Posted 3 years ago #

    Pretty!

    I see - no robots info. I'm guessing again, but I think it only appears when you need to tell the bots something, i.e. to follow but not index, or vice versa. If the command is follow,index, then it doesn't appear.

    To see if that's the case, try no-indexing a page or post as a test. Then see if the command shows up.

    Maybe Joost can confirm at some point...

  10. shnozolla
    Member
    Posted 3 years ago #

    Thank you Miriam for your tips (and for the compliment ;) )

    You are right...the robots tags only appear on the pages that are not meant to be followed.

    I just checked the source code for

    http://www.strawberryswan.com / wp-login

    and the no follow robots appear there!

    I remember reading somewhere that having a robots.txt file increases your search engine rankings...I guess that has changed, as you wrote previously.

    Thank you!

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic