Support » Plugin: All in One SEO » My site got de-indexed

  • Hey devs at AIO, you should be ashamed. I updated the plugin after several weeks thinking that the issue might have been fixed but my whole site got de-indexed. It’s showing that robots.txt is blocking my URLs. My traffic dropped suddenly and now I installed the previous version. What are you doing with your life?

    • This topic was modified 1 month, 1 week ago by sarfrazk9.
Viewing 5 replies - 1 through 5 (of 5 total)
  • Plugin Support arnaudbroes

    (@arnaudbroes)

    Hey @sarfrazk9,

    I just had a look at your website, saw that you downgraded to the previous version like you said and then checked your robots.txt file.

    You still have a disallow / rule in your robots.txt file that is effectively blocking access to every user agent, including search engines. That means the problem wasn’t introduced by the new version of the plugin but was simply carried over when you updated. It’s a pre-existing rule.

    I would recommend you to remove that rule immediately so that search engines can start indexing your website again. There’s no way the plugin added that rule on its own so I would make sure that no other plugins are hooking into the robots.txt file.

    – Arnaud

    Thread Starter sarfrazk9

    (@sarfrazk9)

    The older version which Google webmaster saw on 27th Feb is this:

    User-agent: *
    Allow: /wp-admin/admin-ajax.php
    Disallow: /wp-admin/
    Disallow: /

    Sitemap: https://www.xtremegaminerd.com/sitemap.xml
    Sitemap: https://www.xtremegaminerd.com/sitemap.rss

    The newer version is this

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Sitemap: https://www.xtremegaminerd.com/sitemap.xml

    As you can see the previous version was when the plugin was updated and now the “Disallow: /” is no longer there which was blocking the URLs from being indexed.

    I hope it should be fine within a few days now but the problem is definitely in the plugin as it is no coincidence that the robots.txt changed just after updating to the newer version.

    Plugin Support arnaudbroes

    (@arnaudbroes)

    @sarfrazk9 You’re correct. I discussed this with a colleague and found that I was in fact looking at a cached version of your site’s robots.txt, which is why I still saw the disallow rule.

    After reviewing our code, I’m still fairly certain this was not caused by our plugin. We did make one significant change though in the new version and that is that we now hook into the official robots.txt filter in WordPress instead of overriding it completely ourself (which is what we did previously). That means other plugins and even your theme can also hook into it.

    If you happened to check the robots.txt rules in the new version of our plugin, did you see any rules there? And if not, would you mind sharing a list of your active plugins and theme with us so we can take a look and see if we can reproduce the issue?

    – Arnaud

    Thread Starter sarfrazk9

    (@sarfrazk9)

    After rolling back to the previous version and then again updating to the newer version. It is confirmed that it is indeed the AIOSEO plugin that added this rule. I just tried and observed this now by updating the plugin and the rule was added automatically as my robots.txt was enabled in the feature manager. My site isn’t able to recover. 70% of my earnings lost this month and google webmaster is not allowing my sitemap to be indexed even though I have changed the robots.txt 15 days ago. Good damage is done to the site👍.

    Plugin Support arnaudbroes

    (@arnaudbroes)

    @sarfrazk9 as I explained earlier, we now use the official robots.txt filter hook, which allows other plugins or your theme to hook into it as well. The rule is not coming from AIOSEO – it’s being very likely added by another plugin or theme. Feel free to contact us here (click the “Complete a Form” button) if you’d like us to troubleshoot which plugin or theme is adding it – https://aioseo.com/contact/. Either way, I’m sorry to hear about the damage that has been caused to your website’s traffic and business.

    – Arnaud

Viewing 5 replies - 1 through 5 (of 5 total)
  • You must be logged in to reply to this review.