Network sites 404 Error on robots.txt (21 posts)

  1. duncmorley
    Posted 4 years ago #

    Hi there

    I have installed the Multisite Robots.txt Manager on my network.

    Whenever I go to a site on the network (ie http://www.networksite.com/robots.txt) I get a 404 error.

    The main site (ID: 1) works as it should (ie http://www.mainsite.com/robots.txt)

    Any ideas?

  2. That's a mapped domain, right?

    What happens if you go to the non-mapped URL?


  3. duncmorley
    Posted 4 years ago #

    It is yes. The network sites are installed in sub directories.

    The mainsite.com/networksite automatically redirects to networksite.com.

    So mainsite.com/networksite/robots.txt redirects to networksite.com/robots.txt which goes to a 404 error page.

  4. Do you have a non-mapped site to check with?

    Also which plugin are you using? Can you link to it?

  5. duncmorley
    Posted 4 years ago #

    I am using Multisite Robots.txt Manager. Please see link below:


    I don't have a non mapped site to check unfortunately.

  6. So make a test one :)

  7. duncmorley
    Posted 4 years ago #

    In fact i did have one domain that wasn't mapped yet. mainsite.com/networksite/robots.txt still 404's.

  8. Okay then, it's not mapped domains :) It doesn't like subfolders!

    Annnnd yeah, looks like we ran into this a year ago.

    http://wordpress.org/extend/plugins/pc-robotstxt/ works.

  9. KGarrett
    Posted 4 years ago #

    I'm having the same issue...I just installed it and it doesn't work - still got a 404 error for the subdirectory's file.

  10. KGarrett - If it's the same plugin, it's apparently a known issue. If pc-robotstxt doesn't work, make a new topic for yourself :)

  11. duncmorley
    Posted 4 years ago #

    pc-robotstxt does also not work. Both the mapped domain and unmapped give me 404 error!

  12. tribalNerd
    Posted 4 years ago #

    Hello everyone...

    Currently the MS Robots.txt Manager Plugin doesn't render the robots.txt file within directories. However, the next release of the plugin renders the robots.txt file within a directory. It should be ready in the next day or two.

    Now.... the robots.txt file shouldn't be in a directory at all. Spiders only read robots.txt files from the root of the Website. This is why WordPress itself, doesn't render the robots.txt file within directories.

    Hope that helps clear up some confusion...


  13. That makes a ton of sense, tribalNerd :) Thanks!

  14. duncmorley
    Posted 4 years ago #

    tribalNerb, you're a ledge! Thanks for the info.

  15. pepe
    Posted 4 years ago #

    @tribalNerd: Any roadmap for the update? I have the same problem and I'd really like the plugin to work. I've got a mapped domain (based on a subfolder network installation) and the robots.txt doesn't render correctly. If I change the URL in the network admin sitemanager, it does render, but that breaks the WordPress admin interface.

  16. JustinRS83
    Posted 4 years ago #

    I'm running a multisite instance with 6 sites, 5 of them are working correctly but one is sending me to a 404 page. Any idea what could cause this?

    And yes they are mapped and when I remove mapping so it's in the subfolder that one site still breaks.

  17. JustinRS83
    Posted 4 years ago #

    I found the solution to the issue for me was to reset my permalinks fyi.

  18. Gawain Lynch
    Posted 3 years ago #

    I don't know if this is of any use to anyone, but I struck this problem also on a domain mapped MS install.

    The problem in my case was that the request for robots.txt wasn't being passed through to WordPress and was being intercepted by Nginx.

    Adding this fixed the problem:

    location = /robots.txt {
    	allow		all;
    	log_not_found	off;
    	access_log	off;
    	try_files $uri $uri/ /index.php?$args;

    I can't imagine Apache would be too different.

  19. Rahul Bansal
    Posted 3 years ago #

    Check if robots.txt file is present physically in root directory and is readable.

    Most SEO plugins including wordpress-seo uses common robots.txtfile across network in multisite

  20. Gawain Lynch
    Posted 3 years ago #

    That as my first thought too Rahul, in my case there were no other physical or virtual robots.txt.

  21. Rahul Bansal
    Posted 3 years ago #

    Hmm. I checked my found a robots.txt in my root dir.

    May be your document-root is not writable by php-process owner.

    A small suggestion. Try creating a empty robots.txt and make it 0777. If your SEO plugins (if you using any) update it then most likely its a permission issue.

    We should avoid requests to php as much as possible.

Topic Closed

This topic has been closed to new replies.

About this Topic