How to prevent direct viewing of the robots.txt file? (3 posts)

  1. Benjamin_SP
    Posted 6 years ago #

    Typing any http://www.domainname.com/robots.txt shows the contents of this file. How can I prevent people from viewing this while still allowing search engines access? I know htaccess and htpasswd files can't be viewed in plain site like this.

  2. whooami
    Posted 6 years ago #

    why on earth do you want to invest time in doing something like that?

    its ridiculous, and while it could be done, it would take LOTS of work tracking down the IPs to allow access to. and even then.. you would end up blocking access to any number of new spiders that obey robots.txt

  3. Benjamin_SP
    Posted 6 years ago #

    The only reason I want to do that (and I wish I didn't have to) is because I've read many sploggers use a blog's sitemap to scrap content. This is why I renamed my sitemap (I use the google sitemap plugin) from the usual sitemap.xml to somethingobscure.xml. But now I'm not sure if search engines will pick up this renamed file as the actual sitemap. Plus I saw on this page:


    that the site map should be defined in the robots.txt file like so:

    Sitemap: http://www.example.com/sitemap.xml

    The problem is, if anyone can see my robots.txt file then renaming the sitemap would make no sense. This is the dilemma, I'm afraid of scrapers so I want to hide viewing of this file.

Topic Closed

This topic has been closed to new replies.

About this Topic