Robots.txt Disallow WP Blog Pages

  • apollo24

    (@apollo24)


    10 years, 4 months ago

    Hi All

    On my wp site I have a blog on a different page rather than homepage.

    Let’s say my posts appear on site.com/blog

    So today I have dozens of posts and my site appears on google with site.com/blog/page/2 site.com/blog/page/3 etc.

    My questions is how can I disallow /page/2 /page/3 /page/4 etc. But to allow /blog

    Is it correct to write code like this

    disallow: /blog/page/

    Will it remain /blog open for google bot?

    Thank you

Viewing 8 replies - 1 through 8 (of 8 total)
  • Justin Greer

    (@justingreerbbi)

    10 years, 4 months ago

    I am not an SEO expert but wouldn’t you want to get as mush traction as you possibly could?

    The links being indexes archive listings. If you do not want to include your archive listing for indexing then you could place a NOFOLLOW in the header.php

    Something like

    if(is_archive()){
 echo '<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">';
}
    Thread Starter apollo24

    (@apollo24)

    10 years, 4 months ago

    Hi justingreerbbi

    Looks like these pages are not configured as archive pages. I use All in One SEO and I checked NOINDEX for archive pages, but these pages are still being crawled by Google.

    Justin Greer

    (@justingreerbbi)

    10 years, 4 months ago

    Yes you are correct. Sorry, since you are using a the page “blog” to list your archive it is considered a “page”

    if(is_page() && is_paged()){
 echo '<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">';
}

    I really recommend not adding a “nofollow, noindex” to this but that is just me.

    Thread Starter apollo24

    (@apollo24)

    10 years, 4 months ago

    Does it mean all my pages within the site will be noindexed, doesn’t?

    Justin Greer

    (@justingreerbbi)

    10 years, 4 months ago

    No

    &&

    is a conditional statement which must match both conditions.

    You could be more specific and do 

    if(is_page('blog') && is_paged()){
 echo '<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">';
}
    Thread Starter apollo24

    (@apollo24)

    10 years, 4 months ago

    Great, I’ll try it.

    Why you think it is not recommended to nonindex these pages? They are pure duplicate content.

    Thanks a lot.

    Justin Greer

    (@justingreerbbi)

    10 years, 4 months ago

    Just my preference. Are you displaying all content in the listings or just excerpt?

    Thread Starter apollo24

    (@apollo24)

    10 years, 4 months ago

    Only excerpt

Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘Robots.txt Disallow WP Blog Pages’ is closed to new replies.