Yoast robots.txt in multisite mixes sitemaps across sites

  • Resolved djwilko12

    (@djwilko12)


    7 months, 2 weeks ago

    Hello Yoast team,

    I’ve just discovered a critical issue with how Yoast generates robots.txt files in WordPress multisite installations.In a multisite setup, Yoast appears to generate a single robots.txt that includes sitemap URLs from multiple sites in the network.

    As a result, each individual site ends up declaring sitemaps that belong to other sites, even when they are on different domains and completely unrelated.

    This is extremely problematic from an SEO and Search Console perspective, and it should not happen by default. Robots.txt and sitemap declarations are site-level signals and should not be mixed across sites.

    This is quite disappointing to find, and I would expect Yoast to prevent this behavior or at least make it very explicit.

    I need to fix this urgently. What is the correct and recommended way to stop Yoast from injecting other sites’ sitemaps into each site’s robots.txt in a multisite environment?

    Thanks

Viewing 15 replies - 1 through 15 (of 15 total)
  • Plugin Support Maybellyne

    (@maybellyne)

    7 months, 2 weeks ago

    Hello @djwilko12

    Thanks for reaching out about therobots.txtfile in a multisite environment. Can you let us know if your subsites are subdomains, subfolders/subdirectories or domain mapping?

    Thread Starter djwilko12

    (@djwilko12)

    7 months, 2 weeks ago

    Hi @maybellyne

    Each subsite in the multisite has its own independent domain.
    The multisite is technically structured using subfolders, but every site is mapped to its own domain and operates as a completely separate website.

    These sites are not related to each other in terms of brand, content or SEO strategy.

    Because of this, the way Yoast currently injects a single robots.txt block with sitemap URLs from multiple sites is affecting us. Each site ends up exposing sitemaps that belong to other, unrelated domains.

    This behavior is unexpected and problematic. At the very least, there should be a clear option to prevent other sites’ sitemaps from being included in each site’s robots.txt in a multisite environment.

    Please let me know how this can be corrected urgently.

    Plugin Support Maybellyne

    (@maybellyne)

    7 months, 1 week ago

    Thanks for the clarification. This is actually a limitation in how WordPress multisites with mapped domains work. WordPress generates a single robots.txt file from the network administration domain. While there is no direct way to dynamically generate a separate robots.txt, we did some research and found that it is possible to generate a separate robots.txt per domain dynamically. Please check this article for reference on how to do this.

    Note that we have not tested the code proposed in the article. We would recommend testing it on a staging site and creating a backup before adding or testing the code.

    Thread Starter djwilko12

    (@djwilko12)

    7 months, 1 week ago

    Hello @maybellyne


    Thanks for the explanation, but I think this misses the core of the issue I’m reporting.
    I understand that WordPress multisite generates a single virtual robots.txt at the network level. That part is clear.
    However, the problematic section is a Yoast-specific block, explicitly marked as such, where Yoast injects Sitemap directives for multiple sites across the network:

    START YOAST BLOCK

    Sitemap: …


    This behavior is not imposed by WordPress core. It is a design decision made by Yoast.
    Even if robots.txt is shared, Yoast could:
    – Avoid injecting sitemaps altogether in multisite.
    – Inject only the current site’s sitemap.
    – Or provide a clear setting to disable sitemap output in robots.txt for multisite installations.

    Right now, none of these options exist, and users are left with cross-site sitemap contamination without warning or control.
    I’m not asking for separate robots.txt files per domain. I’m asking how to remove or disable this Yoast-generated sitemap block, or whether Yoast plans to offer a supported way to control this behavior.

    Thanks!

    [ Please do not bump. ]

    Plugin Support Maybellyne

    (@maybellyne)

    7 months ago

    Thanks for suggesting a new feature for our plugin. Please submit feature requests to our GitHub repository for Yoast SEO. You will need an account to create a new issue for your feature request. Please provide as much information about the issue, including the URL to this conversation.

    Thread Starter djwilko12

    (@djwilko12)

    7 months ago

    I’m not suggesting a feature, I’m reporting an issue that your plugin is causing on multisite installs.
    What I need from you is to handle this internally and fix it.
    I don’t want to suggest a feature on GitHub. This is something that’s not working correctly on your side, and I would appreciate it if you could find a solution as soon as possible.
    Thanks.

    Plugin Support Maybellyne

    (@maybellyne)

    7 months ago

    If you consider it a bug, you can also report this on GitHub. Our product team is actively using the bug tracking on our GitHub repository, so your best next step would be to create a new issue for our developers here.

    Thread Starter djwilko12

    (@djwilko12)

    7 months ago

    Hello @maybellyne,

    I don’t think it’s appropriate to simply pass this elsewhere and wash your hands of it. I don’t understand why you keep insisting that I report this somewhere else.

    This is the official support channel for your plugin. I have already documented the issue clearly here, and I will not duplicate the report on another platform.

    If this needs to be escalated, that should be handled internally by your team. That’s precisely what this support space is for.

    The issue is affecting my multisite installations and needs to be addressed. Please escalate it internally, investigate it, and let me know once there is a resolution.

    Thank you.

    Plugin Support Maybellyne

    (@maybellyne)

    7 months ago

    This forum is for providing support for the basic use and setup of the plugin. If you consider an issue a bug, our product development team manages the GitHub repository where you should report this, not the forum. You have the opportunity to provide context and get updates on your report there.

    Torsten Landsiedel

    (@zodiac1978)

    6 months ago

    Hi @djwilko12

    I think, this snippet could solve your issue:

    add_filter( 'wpseo_should_add_subdirectory_multisite_xml_sitemaps', '__return_false' );
    Thread Starter djwilko12

    (@djwilko12)

    6 months ago

    Thank you @zodiac1978 i will check!

    Torsten Landsiedel

    (@zodiac1978)

    6 months ago

    Unfortunately only for Subfolder installs, because of this check:
    https://github.com/Yoast/wordpress-seo/blob/eb827e24ff1d4d5ac660038d25a2c56191b266ec/src/integrations/front-end/robots-txt-integration.php#L184

    Thread Starter djwilko12

    (@djwilko12)

    6 months ago

    In my install, each subsite in the multisite has its own independent domain.
    The multisite is technically structured using subfolders, but every site is mapped to its own domain and operates as a completely separate website. Do you think your solution doesnt fit my structure?

    Torsten Landsiedel

    (@zodiac1978)

    6 months ago

    @djwilko12 It is just one line: just check it out.

    If the multisite is set up as using subfolders, it should work. The check is for is_subdomain_install Fingers crossed!

    Thread Starter djwilko12

    (@djwilko12)

    6 months ago

    Thank you!!

Viewing 15 replies - 1 through 15 (of 15 total)

The topic ‘Yoast robots.txt in multisite mixes sitemaps across sites’ is closed to new replies.