Support » Plugin: The SEO Framework » Comment pagination (pages 2 and above) set to noindex

  • Resolved jumbo

    (@jumbo)


    Hi. I’m looking to transition from Yoast to your plugin. It looks quite impressive.

    I’ve noticed that comment pagination is set to robots noindex even if Apply noindex to every second or later archive page? is checked or unchecked in the settings.

    I’m talking about these types of URLs (page 2 and above):

    /blog/the-title/comment-page-2/
    /blog/the-title/comment-page-3/

    Is this intentional? This is different from Yoast, which does not set such pages to noindex. Instead, Yoast sets the canonical tag to point to the first page while allowing the pages to remain indexed.

    If this is intentional, could you please explain the rationale?

    Thanks.

Viewing 5 replies - 1 through 5 (of 5 total)
  • Plugin Author Sybre Waaijer

    (@cybr)

    Hello!

    Yes, this is intentional. Instead of using a custom canonical URL generator, we rely on WordPress to send us a proper canonical URL via wp_get_canonical_url(). That function uses get_comments_pagenum_link(), which returns different canonical URLs for the comment pages, something Yoast SEO doesn’t do.

    So, we had to use other means of blocking pagination for comments. In our case, we block it via the robots’ noindex property.

    Please keep in mind that when you point the canonical URL to a different page, e.g., from a comment page to the main blog page, you effectively block indexing as well. In other words: We enforce the same rules as Yoast SEO with functionality more close to the WordPress Core API.

    Now, WordPress archives aren’t the same as WordPress comment pages. So, the “Apply noindex to every second or later archive page?” option does not affect comment pagination. We have never considered allowing comment pages to be indexed (nor has Team Yoast), so we don’t have a straightforward switch for that.

    I hope this clears everything up! Cheers πŸ™‚

    We have never considered allowing comment pages to be indexed (nor has Team Yoast), so we don’t have a straightforward switch for that.

    @cybr Thanks for replying. Unless I’m misunderstanding, it seems you’re saying you’re reaching the same end goal, but going about it in different ways. Your plugin does noindex comments pagination as a hard-coded, unchangeable setting, but Yoast doesn’t.

    How I reproduced this:

    1. I did a fresh WordPress install.
      • Installed and activated The SEO Framework plugin.
      • Made no settings changes.
      • Modified WordPress core discussion settings to “Break comments into pages with
        2 top level comments per page
      • Added 6 comments to a post

    On these pages:

    /blog/the-title/comment-page-2/
    /blog/the-title/comment-page-3/

    There is no canonical tag and you’re setting the robots tag to noindex. But, with Yoast they’re not setting noindex on paginated comments pages and they are setting a canonical tag to point to the following primary page:

    /blog/the-title/

    Your approach does seem to be distinct among other SEO plugins. I’m not saying your approach is wrong, but just found the distinction interesting that you’re taking this approach when others are just using canonical to point to the primary page.

    Jumping back to a previous comment you made:

    We enforce the same rules as Yoast SEO with functionality more close to the WordPress Core API.

    When I disable all SEO plugins, the default WordPress behavior with Twenty Twenty theme is to do something different altogether… On the page /blog/the-title/comment-page-2/

    It has the following head tags:

    <link rel=’prev’ href=’/blog/A-SEPARATE-BLOG-POST-IN-SEQUENCE/’ />
    <link rel=”canonical” href=”/blog/the-title/comment-page-2/#comments” />

    So default WordPress is 1) allowing commment pagination pages to be indexed and 2) setting the canonical to the full URL of the comment page and not the primary.

    When I saw how you were implementing this, I thought this made sense, but I’m concerned I’ll end up shooting myself in the foot applying a site-wide change like this without understanding the repercussions, especially since this is a hard-coded configuration in your plugin.

    Thanks.

    Plugin Author Sybre Waaijer

    (@cybr)

    Hi again!

    Yes, the removal of the canonical URL is a side effect of another feature, which I oversaw in my reply earlier. Whenever a “noindex” tag is found, the canonical URL is cleared. We do this because Google otherwise takes months to deindex pages. See https://github.com/sybrew/the-seo-framework/issues/370.

    Google also doesn’t listen to link rel next/prev tags, which is a bug in their system. Because of this, Google is slow to recognize whether comment pages are paginated versions of the main page. So, the comment pages add unwanted and unnecessary weight to your website, among bringing duplicated content issues.

    So, to trickle down the behavior:

    1. Unlike the other SEO plugins you inspected, we use WordPress’ canonical URL generator. This adds the unwanted comment pagination.
      • Thanks to bugs in Google’s parser not honoring prev/next relationships links, comment pagination harms your crawling budget and cause duplicated content issues, and so these pages are unwanted. Therefore, we automatically apply noindex to paginated comment pages.
      • Because Google is slow to deindex pages that have canonical URLs, we automatically remove the canonical URL from pages with noindex applies.
      • Ultimately, we don’t output any canonical URL on comment pagination pages.

    Lastly, because WordPress allows content-pagination via other means, i.e. via the <!--nextpage--> tag, it confuses systems on what’s the pagination intended for. It’s a bit of a mess.

    I hope this clears that up πŸ™‚ If you have more questions, let me know!

    Thanks @cybr for the detailed explanation. I understand your rationale now.

    Plugin Author Sybre Waaijer

    (@cybr)

    Hello again!

    In the next update (TSF v4.0.5), you can use this simplified and use-case-appropriate filter. It will disable the noindex tag set on comment-paginated pages:
    add_filter( 'the_seo_framework_enable_noindex_comment_pagination', '__return_false' );

    However, I do not recommend using it as it’ll likely get those pages flagged as duplicated by Google anyway; unless your theme strips or trims the content.

Viewing 5 replies - 1 through 5 (of 5 total)
  • You must be logged in to reply to this topic.