Support » Plugin: Multisite Robots.txt Manager » mapped domain robots.txt 404

  • Resolved sslone


    I’m having an issue with the robot txt file not being found for 2 of the 3 sites I’m running in WordPress Network – all 3 are mapped domains.

    I’m running ‘WordPress MU Domain Mapping‘ for the mapping, and & ‘BWP Google XML Sitemaps‘ for site maps.

    The robots.txt is displaying for the one of the three mapped domains, but the other two are returning 404s for the mapped robots file – but will display it for the subdirectory path.

    Any ideas?

Viewing 13 replies - 1 through 13 (of 13 total)
  • BTW, your link on Multisite Robots.txt Manager – Network Settings’ page to BWP google XML Sitemaps goes to ‘Google (XML) Sitemaps Generator for WordPress’ — not Better WordPress Google XML Sitemaps (BWP Google XML Sitemaps) .

    Plugin Author tribalNerd


    Hello sslone,
    This issue can sometimes happen when a different robots.txt manager was previously used. The other plugins leave a robots.txt file in the Websites settings (network admin > sites > edit site > settings) – if this is the issue, locate the old robots.txt file in the settings, make it blank, then save your settings.

    My plugins robots.txt setting name is: MS Robots.txt – make sure all other robots.txt files are blank, other than the MS one.

    ….. I’ll correct the links, thanks.

    Let me know if this helps ~Chris


    We’re experiencing the same issue : the default robots.txt is working but any other one (using WordPress MU Domain Mapping) is not.
    It ends up in 404.

    I’ve checked the settings, there’s nothing but “Ms Robotstxt” in the sites settings.

    The default site has however another setting that is “Ms Robotstxt default”.

    I actually had to go into the database tables for each site and make sure that the robots.txt was enabled – there was a difference for the sites that correctly displayed the robots.txt file, and the ones that did not. I didn’t record the table or field that needed tweaking, though, but i believe it was in options table.

    There’s not a lot of fields in the option table.

    The “autoload” field is set to “yes” on the working/default site…
    While it’s set to “no” on the non working sites.

    I’ve just tried to put “yes” instead of “no”…
    Not working either, still 404.

    Still not working 8(

    Actually it seems what should not be working (sub-folder sites) is working while domain-mapped sites are not.

    http://wp-mu.domain.tld/site/robots.txt works and contains the correct data.

    http://www.domain-mapped-site.tld/robots.txt gives a 404.

    We’re using this domain mapping plugin :

    WordPress MU Domain Mapping

    Plugin Author tribalNerd


    Hello networkstudio,

    I will be releasing a new version of the plugin later today or first thing tomorrow morning. The new version removes the path check and doesn’t care if domains are mapped or not. It should correct the issues you’re having.


    Any update on the new version? I’ve got site that was working and now is no longer resolving – getting the 404 error again.

    i actually went in to the options table (rewrite_rules) and added the text below – this fixed the issue for the single site that was delivering a 404 for the robots.txt


    Plugin Author tribalNerd


    Hey sslone…
    I thought it would be ready this morning but I ran into a small bug. I’m just doing some final testing and updating help documents. It should be ready Thursday morning.

    Any idea how that rewrite rule was removed?

    I will write function to detect the missing rewrite rule and a button to click that will add the rule back if it’s missing.


    This is _exactly_ our issue…

    The rewrite rule is not added when a new site/blog is created.

    I’ve just added it manually (once per site/blog) and it now works.

    Plugin Author tribalNerd


    Quick update… I’ve created a function that detects and corrects missing rules and/or the missing robots.txt line. As well, a function that detects/removes robots.txt file data from old plugins.

    And finally, I’m still testing the newest version. It should be ready today (and yesterday, and the day before), assuming I don’t find any more pain in the butt bugs.


    Plugin Author tribalNerd


    Hello again…

    Thanks sslone for finding that rather odd problem.

    The newly released version now detects the missing rewrite rule and it looks for old robots.txt plugin data. You can also manually run the checks via the Help tab. When an issue is found, an info box with buttons will display, allowing you to correct the issue.

    Let me know if you have any issues with the new feature.


Viewing 13 replies - 1 through 13 (of 13 total)
  • The topic ‘mapped domain robots.txt 404’ is closed to new replies.