WordPress.org

Ready to get started?Download WordPress

Forums

Search engine indexing problem (2 posts)

  1. Will Taft
    Member
    Posted 7 years ago #

    I was told search engines don't like duplicate content, well I have quadruple content! Posts from my blog show up multiple times in Google search results. For example a single post will be indexed as:

    1.) mysite.com/wordpress/index.php?p=44
    2.) mysite.com/blog/
    3.) mysite.com/archives/
    4.) mysite.com/the-category-name/the-post-name/

    It seems like the only one I would want indexed is #4 which is how the url is for each single post page. Is that correct?

    Or would it be #2, which is the whole blog? The issue with #2 is that at /blog, older posts are no longer on the page.

    With advice from another thread, I put a simple robots.txt file on my site to do this:
    User-agent: *
    Disallow: /wp-register.php
    Disallow: /wp-login.php

    I found the following recommended robots.txt file at askapache.com. Does anyone see any problem with adding its content to my robots.txt file?

    Will it work or is it overkill? It gets confusing for me, but one potential problem I see is the following robots.txt file disallows "/category/" but because all my single page posts have a url like #4, above, I probably should not disallow "/category/"? And trackbacks for posts are the same as #4 with "/trackback/" appended.

    -----------------------------
    Apache recommended robots.txt for WordPress 2.+

    http://wordpress.pastebin.ca/552045

    Thanks.

  2. Will Taft
    Member
    Posted 7 years ago #

    Has anyone had a chance to look at the paste bin at the Apache WP file? It seems to cover most anything you might want to block on a WP blog, but I think it would still allow indexing of my single posts, (#4 above).

    But..... I did read that the Google bot looks only at the first user agent entry it comes to so if that is true, the Apache file would not work because the Google bot would follow the "user agent *" and not look any further down to "user agent Googlebot"

    Can anyone help clarify this?

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags