WordPress.org

Ready to get started?Download WordPress

Forums

How to direct bots to correct files? (7 posts)

  1. randybrown
    Member
    Posted 9 years ago #

    Lookign through my server logs, I see a lot of requests for index.rdf and rss.xml. I'm sure these are coming from some search bot.
    Can anyone give me details on what I might do to my robots.txt file or a meta entry to direct these requests to the appropriate WP files?
    TIA

  2. OperaManiac
    Member
    Posted 9 years ago #

    wont copying wp-rss.php as rss.xml work?

  3. Mark (podz)
    Support Maven
    Posted 9 years ago #

    I have search bots looking for files which just do NOT exist on my site ..... not so much a case of just indexing the contents, it seems they have their own agenda too.
    Bad enough they steal the content you do have, it would seem they are also 'ripping open the mattresses'.

  4. raena
    Member
    Posted 9 years ago #

    Naming feeds with filenames like index.rdf and rss.xml is a common habit, so the bot is probably just guessing.
    If you like, you can use an Apache .htaccess redirect to tell them where they can really be found, like so:
    Redirect permanent /index.xml /wp-rss.php
    Redirect permanent /rss.xml /wp-rss.php
    A good and polite bot will get the hint from the 'permanent' directive. A crappy bot will not get the hint and will keep guessing, but it should still find the page anyway.

  5. randybrown
    Member
    Posted 9 years ago #

    raena - I tried your suggestion. Placed those two lines at the bottom of my .htaccess file. But it caused 500 errors.
    Could there be another method, or a typo?
    TIA

  6. Stevarino
    Member
    Posted 9 years ago #

    hm, what about mod rewrite? not the best solution but definately a quick fix if redirects arn't working.

  7. OperaManiac
    Member
    Posted 9 years ago #

    the reason for google searching for this files is rumored to be becoz google is apparently planning for a blog search engine... :)

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags

No tags yet.