Support » Fixing WordPress » Search engine indexing problem

  • I was told search engines don’t like duplicate content, well I have quadruple content! Posts from my blog show up multiple times in Google search results. For example a single post will be indexed as:


    It seems like the only one I would want indexed is #4 which is how the url is for each single post page. Is that correct?

    Or would it be #2, which is the whole blog? The issue with #2 is that at /blog, older posts are no longer on the page.

    With advice from another thread, I put a simple robots.txt file on my site to do this:
    User-agent: *
    Disallow: /wp-register.php
    Disallow: /wp-login.php

    I found the following recommended robots.txt file at Does anyone see any problem with adding its content to my robots.txt file?

    Will it work or is it overkill? It gets confusing for me, but one potential problem I see is the following robots.txt file disallows “/category/” but because all my single page posts have a url like #4, above, I probably should not disallow “/category/”? And trackbacks for posts are the same as #4 with “/trackback/” appended.

    Apache recommended robots.txt for WordPress 2.+


Viewing 1 replies (of 1 total)
  • Has anyone had a chance to look at the paste bin at the Apache WP file? It seems to cover most anything you might want to block on a WP blog, but I think it would still allow indexing of my single posts, (#4 above).

    But….. I did read that the Google bot looks only at the first user agent entry it comes to so if that is true, the Apache file would not work because the Google bot would follow the “user agent *” and not look any further down to “user agent Googlebot”

    Can anyone help clarify this?

Viewing 1 replies (of 1 total)
  • The topic ‘Search engine indexing problem’ is closed to new replies.