Support » Plugin: All In One WP Security & Firewall » “Block Fake Googlebots” feature blocking REAL Google bots

  • Hello,

    For a couple of days I was noticing that whenever I try to submit my sitemap xml file to Google, it generates an error as “no links in the submitted sitemap”.

    For a solution i Searched and found a post having the same problem. Now when I switch off the feature. Google starts crawling and everything seems good.

    If you please see into the matter, why this feature is blocking the real google bots. I have turned off the feature for now, till there is a solution.

    Kind regards,

Viewing 13 replies - 1 through 13 (of 13 total)
  • Plugin Contributor mbrsolution

    (@mbrsolution)

    Hi, which sitemap plugin are you using?

    I have this plugin installed in many sites with the feature you mentioned enabled and the sitemap is read correctly by Google.

    Kind regards

    Thread Starter svayam

    (@svayam)

    Hi,

    I am using Yoast SEO v 5.7 and it’s inbuilt sitemap.

    Kind regards,

    Plugin Contributor mbrsolution

    (@mbrsolution)

    Hi, I am also using Yoast SEO v 5.8 with WordPress 4.8.3. I am able to submit the sitemap without any issues.

    There must be something else blocking this in your site. Check the log files in AIOWPS and your server log files. Have you tried disabling all your other plugins except Yoast and AIOWPS, then carrying out a test?

    Kind regards

    • This reply was modified 4 years, 2 months ago by mbrsolution.
    Thread Starter svayam

    (@svayam)

    Hi,

    Thanks for the suggestion. I have triple checked all the settings and disabled all plugins, and everything possible from my end. One important thing, don’t know if it’s related with this issue, still I am mentioning here, I have blocked certain Bad IP’s which were trying to find my “Default WP Login Page”.

    So I removed all of them in case if there may be any problem. But still no luck. Just to know, when I am blacklisting some IP it shows blacklisted in the 404 error log but they don’t show up on the “Permanent Blocked IP” tab-dashboard. Why this is so? Are they different?

    Anyway that’s all information I can provide you. It just simply works if I turn off the feature. Really weird. Thanks again.

    Kind regards,

    Plugin Contributor mbrsolution

    (@mbrsolution)

    Hi in relation to the following comment. How are you adding your black listed IP addresses? Are you adding them to the following Blacklist Manager menu?

    when I am blacklisting some IP it shows blacklisted in the 404 error log but they don’t show up on the “Permanent Blocked IP” tab-dashboard. Why this is so? Are they different?

    I am not sure why this is happening in your site. There must be some other issue that is causing your sitemap to fail in Google Webmasters Tools. Are you using a CDN service? Would you be able to try a different sitemap plugin Google XML Sitemaps just for testing purposes?

    Regards

    Plugin Contributor mbrsolution

    (@mbrsolution)

    Hi, I just remembered another user with a similar issue to yours. Please read the following thread.

    This might bring some light to your issue.

    Kind regards

    Thread Starter svayam

    (@svayam)

    Hi,

    First of all thank you for the reference link, I have carefully gone through with multiple tests on my site all over the day. I forgot to mention that my site is using CDN by Cloudflare. Even I also found certain articles similar to that.

    1) I white-listed google ip/ip ranges, ASN details everything but just one thing which is not working is AIOWPS “Internet Bot Settings” settings. When turned off simply works…turned on Block. I am not a technical expert obviously to deal with this issue, but I have gone through as my articles that can be found on this issue. But couldn’t find a solution. I am just an advanced user, that’s all. Is there any possible way of exclusion or something like that through htaccess or just have to keep it off?

    2) Now, I block those specific IP’s using the Blacklist Manager, isn’t that a right way? Am I doing something wrong here?

    3) Please Suggest if Possible – I am using Cloudflare mainly for two reasons, a: DDoS protection, b: WAF. There is just 1 another plugin “Ninja Firewall” offering WAF, I thought to use with AIOWPS but am not sure as it may cause conflict. I read regarding this on the support portal both on AIOWPS and Ninja Firewall. So these are the only free option I am left with. If you could suggest anything on this, will definitely be useful.

    Thank you so much for helping.

    Kind regards,

    Plugin Contributor mbrsolution

    (@mbrsolution)

    Hi, as far as I know you are doing everything correctly. I am out of ideas to further troubleshoot and help you with this issue.

    I have submitted a message to the developers to investigate further your issue.

    Sorry about this.

    Kind regards

    Plugin Contributor wpsolutions

    (@wpsolutions)

    Hi @svayam,
    Are you able to consistently reproduce this issue when using the “Fetch as Google” feature in the Google webmaster tools?
    ie, when you have the “Block Fake Googlebots” feature active and you do a “Fetch as Google” operation does it fail?

    Thread Starter svayam

    (@svayam)

    Hi,

    @mbrsolution I understand this weird behaviour, thanks for submitting the issue to the developers. We all will be benefited from the solution.

    @wpsolutions, Yes I tried Fetch as Google with Fake Google Bot feature turned on, which is working fine! And again I tried to see if it’s reading the robots text or not it shows blocked by robots. So that part is fine.

    Let me mention another error which I noticed earlier but couldn’t connect. That this same feature is also blocking the “Google Analytics Dashboard for WP (GADWP)” Plugin as well. I thought it was the plugin problem, but when i disable that particular feature it works fine. I also raised an issue at their forum without knowing. GADWP-Link. I need to tell them as well, it’s not their mistake. Anyways…

    So this is all info i can provide for now. Please let me know if I can help you with anymore data. Thanks.

    Kind regards,

    Neo_II

    (@neo_ii)

    Hello!

    I have the same problem and can narrow it down to one setting:
    Block Fake Googlebots ON: Google can’t read the blog’s sitemap
    Block Fake Googlebots OFF: Everything’s fine.

    So it seems as if the setting is actually blocking the REAL Google bot from accessing the page.

    Cheers,
    Mario

    Plugin Contributor wpsolutions

    (@wpsolutions)

    Hi @svayam,
    Did you try changing the IP address settings to see if that makes a difference?
    I’m referring to the following settings:
    Settings >> Advanced Settings >> IP Retrieval Settings

    Sometimes people have a hosting setup where there might be load-balancers or proxies and retrieving the IP address via the $_SERVER[‘REMOTE_ADDR’] gives the proxy address instead of the real visitor IP address.
    Try selecting “HTTP_X_FORWARDED_FOR” and see if that improves the situation.

    Thread Starter svayam

    (@svayam)

    Hi,

    What a wonderful solution! Just smooth as butter. Truly thankful to you and your team. It’s working like a charm! Thanks once again.

    I am marking this issue as solved. 🙂

Viewing 13 replies - 1 through 13 (of 13 total)
  • The topic ‘“Block Fake Googlebots” feature blocking REAL Google bots’ is closed to new replies.