How to disable the automatically generated robots.txt? (2 posts)

  1. bloup
    Posted 4 years ago #

    I would like no robots.txt at all. How can i properly disable the automatically generated robots.txt from wordpress ?

    I already deleted the do_robots function in /wp-includes/functions.php, nevertheless, i have a php error when i go to revspa.com/robots.txt

    I d like to get a clean 404 error.

    thank you

  2. bloup
    Posted 4 years ago #

    I have found a solution myself, without modifying functions.php
    You just need to add
    redirect 404 /robots.txt to .htaccess and then remove the /robots.txt using URL removal tool from google webmaster tools.

    I am doing so because i think robots.txt is counter-productive. I noticed i lost traffic each time i added disallowed folders to robots.txt
    I don't have documents to hide.

    On the other hand, matt cutt said we should not block folders where there is .js
    Then, .swf are not indexed by wordpress.
    So i think it is useless to block /wp-includes/ or /wp-content/

    At last, i think it is useless to block /wp-admin/ and /wp-includes/ because spam bots don't read robots.txt anyways.

Topic Closed

This topic has been closed to new replies.

About this Topic