Typing any http://www.domainname.com/robots.txt shows the contents of this file. How can I prevent people from viewing this while still allowing search engines access? I know htaccess and htpasswd files can't be viewed in plain site like this.
How to prevent direct viewing of the robots.txt file? (3 posts)
-
Benjamin_SP
Posted 2 months ago # -
why on earth do you want to invest time in doing something like that?
its ridiculous, and while it could be done, it would take LOTS of work tracking down the IPs to allow access to. and even then.. you would end up blocking access to any number of new spiders that obey robots.txt
-
Posted 2 months ago #The only reason I want to do that (and I wish I didn't have to) is because I've read many sploggers use a blog's sitemap to scrap content. This is why I renamed my sitemap (I use the google sitemap plugin) from the usual sitemap.xml to somethingobscure.xml. But now I'm not sure if search engines will pick up this renamed file as the actual sitemap. Plus I saw on this page:
http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress
that the site map should be defined in the robots.txt file like so:
Sitemap: http://www.example.com/sitemap.xml
The problem is, if anyone can see my robots.txt file then renaming the sitemap would make no sense. This is the dilemma, I'm afraid of scrapers so I want to hide viewing of this file.
Reply
You must log in to post.
