Posted 1 year ago #
Hi there
I have installed the Multisite Robots.txt Manager on my network.
Whenever I go to a site on the network (ie http://www.networksite.com/robots.txt) I get a 404 error.
The main site (ID: 1) works as it should (ie http://www.mainsite.com/robots.txt)
Any ideas?
Posted 1 year ago #
That's a mapped domain, right?
What happens if you go to the non-mapped URL?
networksite.mainsite.com/robots.txt
Posted 1 year ago #
It is yes. The network sites are installed in sub directories.
The mainsite.com/networksite automatically redirects to networksite.com.
So mainsite.com/networksite/robots.txt redirects to networksite.com/robots.txt which goes to a 404 error page.
Posted 1 year ago #
Do you have a non-mapped site to check with?
Also which plugin are you using? Can you link to it?
Posted 1 year ago #
I am using Multisite Robots.txt Manager. Please see link below:
http://wordpress.org/extend/plugins/multisite-robotstxt-manager/
I don't have a non mapped site to check unfortunately.
Posted 1 year ago #
So make a test one :)
Posted 1 year ago #
In fact i did have one domain that wasn't mapped yet. mainsite.com/networksite/robots.txt still 404's.
Posted 1 year ago #
Okay then, it's not mapped domains :) It doesn't like subfolders!
Annnnd yeah, looks like we ran into this a year ago.
Posted 1 year ago #
I'm having the same issue...I just installed it and it doesn't work - still got a 404 error for the subdirectory's file.
Posted 1 year ago #
KGarrett - If it's the same plugin, it's apparently a known issue. If pc-robotstxt doesn't work, make a new topic for yourself :)
Posted 1 year ago #
pc-robotstxt does also not work. Both the mapped domain and unmapped give me 404 error!
Posted 1 year ago #
Hello everyone...
Currently the MS Robots.txt Manager Plugin doesn't render the robots.txt file within directories. However, the next release of the plugin renders the robots.txt file within a directory. It should be ready in the next day or two.
Now.... the robots.txt file shouldn't be in a directory at all. Spiders only read robots.txt files from the root of the Website. This is why WordPress itself, doesn't render the robots.txt file within directories.
Hope that helps clear up some confusion...
~Chris
Posted 1 year ago #
That makes a ton of sense, tribalNerd :) Thanks!
Posted 1 year ago #
tribalNerb, you're a ledge! Thanks for the info.
Posted 11 months ago #
@tribalNerd: Any roadmap for the update? I have the same problem and I'd really like the plugin to work. I've got a mapped domain (based on a subfolder network installation) and the robots.txt doesn't render correctly. If I change the URL in the network admin sitemanager, it does render, but that breaks the WordPress admin interface.
Posted 11 months ago #
I'm running a multisite instance with 6 sites, 5 of them are working correctly but one is sending me to a 404 page. Any idea what could cause this?
And yes they are mapped and when I remove mapping so it's in the subfolder that one site still breaks.
Posted 11 months ago #
I found the solution to the issue for me was to reset my permalinks fyi.
Posted 7 months ago #
I don't know if this is of any use to anyone, but I struck this problem also on a domain mapped MS install.
The problem in my case was that the request for robots.txt wasn't being passed through to WordPress and was being intercepted by Nginx.
Adding this fixed the problem:
location = /robots.txt {
allow all;
log_not_found off;
access_log off;
try_files $uri $uri/ /index.php?$args;
}I can't imagine Apache would be too different.
Check if robots.txt file is present physically in root directory and is readable.
Most SEO plugins including wordpress-seo uses common robots.txtfile across network in multisite
Posted 7 months ago #
That as my first thought too Rahul, in my case there were no other physical or virtual robots.txt.
Hmm. I checked my found a robots.txt in my root dir.
May be your document-root is not writable by php-process owner.
A small suggestion. Try creating a empty robots.txt and make it 0777. If your SEO plugins (if you using any) update it then most likely its a permission issue.
We should avoid requests to php as much as possible.
This topic has been closed to new replies.
