WordPress.org

Ready to get started?Download WordPress

Forums

Multisite Robots.txt Manager
[resolved] mapped domain robots.txt 404 (14 posts)

  1. sslone
    Member
    Posted 1 year ago #

    I'm having an issue with the robot txt file not being found for 2 of the 3 sites I'm running in WordPress Network - all 3 are mapped domains.

    I'm running 'WordPress MU Domain Mapping' for the mapping, and & 'BWP Google XML Sitemaps' for site maps.

    The robots.txt is displaying for the one of the three mapped domains, but the other two are returning 404s for the mapped robots file - but will display it for the subdirectory path.

    Any ideas?

    http://wordpress.org/extend/plugins/multisite-robotstxt-manager/

  2. sslone
    Member
    Posted 1 year ago #

    BTW, your link on Multisite Robots.txt Manager - Network Settings' page to BWP google XML Sitemaps goes to 'Google (XML) Sitemaps Generator for WordPress' -- not Better WordPress Google XML Sitemaps (BWP Google XML Sitemaps) .

  3. tribalNerd
    Member
    Plugin Author

    Posted 1 year ago #

    Hello sslone,
    This issue can sometimes happen when a different robots.txt manager was previously used. The other plugins leave a robots.txt file in the Websites settings (network admin > sites > edit site > settings) - if this is the issue, locate the old robots.txt file in the settings, make it blank, then save your settings.

    My plugins robots.txt setting name is: MS Robots.txt - make sure all other robots.txt files are blank, other than the MS one.

    ..... I'll correct the links, thanks.

    Let me know if this helps ~Chris

  4. networkstudio
    Member
    Posted 1 year ago #

    Hi.

    We're experiencing the same issue : the default robots.txt is working but any other one (using WordPress MU Domain Mapping) is not.
    It ends up in 404.

    I've checked the settings, there's nothing but "Ms Robotstxt" in the sites settings.

    The default site has however another setting that is "Ms Robotstxt default".

  5. sslone
    Member
    Posted 1 year ago #

    I actually had to go into the database tables for each site and make sure that the robots.txt was enabled - there was a difference for the sites that correctly displayed the robots.txt file, and the ones that did not. I didn't record the table or field that needed tweaking, though, but i believe it was in options table.

  6. networkstudio
    Member
    Posted 1 year ago #

    There's not a lot of fields in the option table.

    The "autoload" field is set to "yes" on the working/default site...
    While it's set to "no" on the non working sites.

    I've just tried to put "yes" instead of "no"...
    Not working either, still 404.

  7. networkstudio
    Member
    Posted 1 year ago #

    Still not working 8(

    Actually it seems what should not be working (sub-folder sites) is working while domain-mapped sites are not.

    http://wp-mu.domain.tld/site/robots.txt works and contains the correct data.

    http://www.domain-mapped-site.tld/robots.txt gives a 404.

    We're using this domain mapping plugin :
    http://ocaoimh.ie/wordpress-mu-domain-mapping/

  8. tribalNerd
    Member
    Plugin Author

    Posted 1 year ago #

    Hello networkstudio,

    I will be releasing a new version of the plugin later today or first thing tomorrow morning. The new version removes the path check and doesn't care if domains are mapped or not. It should correct the issues you're having.

    ~Chris

  9. sslone
    Member
    Posted 1 year ago #

    Any update on the new version? I've got site that was working and now is no longer resolving - getting the 404 error again.

  10. sslone
    Member
    Posted 1 year ago #

    i actually went in to the options table (rewrite_rules) and added the text below - this fixed the issue for the single site that was delivering a 404 for the robots.txt

    s:12:"robots\.txt$";s:18:"index.php?robots=1";

  11. tribalNerd
    Member
    Plugin Author

    Posted 1 year ago #

    Hey sslone...
    I thought it would be ready this morning but I ran into a small bug. I'm just doing some final testing and updating help documents. It should be ready Thursday morning.

    Any idea how that rewrite rule was removed?

    I will write function to detect the missing rewrite rule and a button to click that will add the rule back if it's missing.

    ~Chris

  12. networkstudio
    Member
    Posted 1 year ago #

    This is _exactly_ our issue...

    The rewrite rule is not added when a new site/blog is created.

    I've just added it manually (once per site/blog) and it now works.

  13. tribalNerd
    Member
    Plugin Author

    Posted 1 year ago #

    Quick update... I've created a function that detects and corrects missing rules and/or the missing robots.txt line. As well, a function that detects/removes robots.txt file data from old plugins.

    And finally, I'm still testing the newest version. It should be ready today (and yesterday, and the day before), assuming I don't find any more pain in the butt bugs.

    ~Chris

  14. tribalNerd
    Member
    Plugin Author

    Posted 1 year ago #

    Hello again...

    Thanks sslone for finding that rather odd problem.

    The newly released version now detects the missing rewrite rule and it looks for old robots.txt plugin data. You can also manually run the checks via the Help tab. When an issue is found, an info box with buttons will display, allowing you to correct the issue.

    Let me know if you have any issues with the new feature.

    ~Chris

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic

Tags

No tags yet.