Again I apologize for the length of the posts however I wanted to give you all a pretty current example of all the good and bad spiders so you wouldn't have to spend the time researching them.
For redirecting to other pages and sites, there are many ways to do this using htaccess. I'll describe a few and let you read apaches manuals for the rest.
Starting with redirecting to new pages in a different directory. For this you will need to put the htaccess file in the same directory you want to be redirected. Make sure you do use this method in /.
RedirectMatch (.*)/oldfile.php
http://www.mysite.com/newdirectory/newfile.php
For multiple redirects, should appear on separate lines.
This changes the directory to where the "new" source files are located.
Maybe you'd like to redirect the entire directory?
RedirectMatch (.*)/
http://www.mysite.com/directory/index.php
Another way to do this and is cleaner:
RewriteEngine on
RewriteBase /~dir/
RewriteRule ^index\.php$ home.php
This adds a little backward compatibility and users wont even know the page has been changed.
or maybe the entire directory?
Redirect /sourcedir http://www.mysite.com/newdir
Your source directory should be the same path as it is right after your docroot.
Alot of times you'll want to capture the query strings and they are NOT a part of the url. You'll need to do something like this:
RewriteCond %{QUERY_STRING} ^cat=([0-9]+)
RewriteRule /index\.php$ http://www.yoursite.com/your page.html [R=301,L]
Of course this is sending the request to a 301 page but you can send it to wherever you like.
This will 'copy' the value of the cat= numeric parameter from the query string into the new URL. You will need to preface this code with Options all or Options +FollowSymLinks.
hmm ok enough ...
Hope that helps