"Your homepage cannot be indexed by search engines. This is very bad for SEO"

  • mvandemar

    (@mvandemar)


    8 years, 5 months ago

    I had a client’s site that was getting this message in the dashboard:

    “Your homepage cannot be indexed by search engines. This is very bad for SEO and should be fixed.”

    This message is actually from the WordPress SEO plugin by Yoast. I searched the forums and found several people who were getting the same issue, and who were able to rectify it by visiting the Yoast widget in the dashboard, and clicking the OnPage.org “Fetch again” button:

    https://wordpress.org/support/topic/updated-and-now-it-says-homepage-cant-be-indexed
    https://wordpress.org/support/topic/your-homepage-cannot-be-indexed-by-search-engines

    However, there are several people who were unable to clear the message, and I found myself in the same situation. I was about to write off the error as erroneous, when I decided to check the headers using several different tools on a couple of different sites. When I did, I discovered that the site was actually returning a 503 Internal Server Error to bots, and when I went through the access logs I realized that it was doing the same to Googlebot as well.

    I disabled all of the plugins and re-tested, and the error went away. I then re-enabled them all aside from Wordfence, and it stayed away, but as soon as I enabled Wordfence all of the bots, including Googlebot, were blocked again:

    74.125.18.224 – – [27/Nov/2015:00:21:06 -0600] “GET /robots.txt HTTP/1.1” 503 2438 “-” “-“
    74.125.18.224 – – [27/Nov/2015:00:21:06 -0600] “GET / HTTP/1.1” 503 2438 “-” “Googlebot/2.1 (+http://www.google.com/bot.html)”

    I verified with ARIN that those are indeed Google IP addresses. I have no idea why Wordfence has started blocking Google IP addresses, but this will kill your seo dead if it is happening to you. Also, there are 0 IPs showing in the “Blocked IP” list, so these appear to be blocked by default and not blocked due to bad behavior.

    After disabling Wordfence I was able to also successfully use the Fetch Again button in the Yoast SEO widget, and OnPage.org was able to verify that the site was visible.

    https://wordpress.org/plugins/wordfence/

Viewing 8 replies - 1 through 8 (of 8 total)
  • alberto14

    (@alberto14)

    8 years, 5 months ago

    Thanks Mvamdemar, I did what you said here and fixed my error, I tried all other recommendations in other forums without any luck, yours was the winning one.I guess Wordfence needs to get this square away because I’m already thinking about other security plugins that wont affect my rankings
    Good job,thanks.

    Plugin Author WFMattR

    (@wfmattr)

    8 years, 4 months ago

    @mvandemar:

    Thank you for the report. These IPs are not blocked by default, but automatic blocks are not permanent, so they may have originally been blocked for breaking one of your firewall rules, but expired before you saw that they were not blocked.

    We use a method to verify Googlebot that is given by Google here:
    https://support.google.com/webmasters/answer/80553?hl=en

    But strangely, if I look up 74.125.18.224, it has no reverse DNS entry as it should. (Just an SOA record, which does show it belongs to Google.)

    If you make your firewall rules less strict, that may be the best solution for now, but I will pass this on to the dev team to check it out.

    -Matt R

    Thread Starter mvandemar

    (@mvandemar)

    8 years, 4 months ago

    @wfmattr – there were 0 ips in the block list, yet Googlebot, my own Bad Neighborhood header detector, http://www.bad-neighborhood.com/header_detector.php, web-sniffer.net, and OnPage.org were all getting 503 errors until I disabled Wordfence.

    Without knowing what was triggering it then disabling Wordfence altogether is the “best” solution, since otherwise it is causing damage to the site.

    -Michael

    Plugin Author WFMattR

    (@wfmattr)

    8 years, 4 months ago

    Ok, I haven’t seen multiple sites blocked with a “503” error, with no blocked IPs ever appearing in the list, so I can’t say what might be the problem without more details — but if you don’t want to troubleshoot since it is a live site, I understand.

    -Matt R

    Thread Starter mvandemar

    (@mvandemar)

    8 years, 4 months ago

    @wfmattr – I can ask the client, or I can see if I can replicate it on a dev site. I am just really busy at the moment. You might want to check with the people who were reporting the same problem in these threads where the normal solutions were not working (Mirella2000, elirnerak, idesign4u1, emh1969)

    https://wordpress.org/support/topic/your-homepage-cannot-be-indexed-by-search-engines

    https://wordpress.org/support/topic/updated-and-now-it-says-homepage-cant-be-indexed

    -Michael

    Plugin Author WFMattR

    (@wfmattr)

    8 years, 4 months ago

    Ok, I’ll watch for your reply if you get a chance to check it out. If I have time, I may check with the others you mentioned, to see if it may be a similar issue too. Thanks!

    -Matt

    Thread Starter mvandemar

    (@mvandemar)

    8 years, 4 months ago

    @wfmattr I have asked the client if we can use her site for debugging and will let you know once she gets back to me. I can tell you 2 things:

    1) If I disable all other plugins on her site aside from Wordfence, it’s still happening.

    2) Everything is up to date

    3) Installed WF on my own site and it does not happen there.

    -Michael

    Plugin Author WFMattR

    (@wfmattr)

    8 years, 4 months ago

    Ok. If you do have a site available to test on (including a dev site, if it is reachable by Google), make sure your email address is entered in the Wordfence options, and “Alert when an IP address is blocked” is enabled. Since blocked IPs expire after a short time (unless you choose a longer period), this is a good way to see when they happened, and the IP address that Wordfence saw.

    If you have Google’s webmaster tools (or “search console”) set up for the client’s site, can you try having it fetch a page, and view what it shows? That should have a message from Wordfence explaining why it is blocked, which might narrow down the issue.

    On the client’s site, to disable most types of blocking, you can turn off the “Enable Firewall” option. This turns off some protection, but will most likely prevent the problem until we can find out what is happening, without disabling Wordfence entirely.

    -Matt R

Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘"Your homepage cannot be indexed by search engines. This is very bad for SEO"’ is closed to new replies.

Tags