Crawl errors related to plugins?

  • xprt007

    (@xprt007)


    7 years, 11 months ago

    Hi there

    I have been having the following crawling errors in Google console, producing response code 500:

    wp-content/plugins/recipepress-reloaded/lib/tgm/class-tgm-plugin-activation.php
    wp-content/plugins/recipepress-reloaded/views/metabox_ingredients.php
    wp-content/plugins/recipepress-reloaded/views/metabox_instructions.php
    wp-content/plugins/recipepress-reloaded/views/metabox_notes.php
    wp-content/plugins/recipepress-reloaded/views/rpr_recipe_public.php
    wp-content/plugins/recipepress-reloaded/widgets/rpr_widget_tag_cloud.php
    wp-content/plugins/recipepress-reloaded/views/admin.php
    wp-content/plugins/recipepress-reloaded/views/metabox_details.php
    wp-content/plugins/recipepress-reloaded/views/settings.php

    I have read in several posts online regarding robots.txt & co, that it’s no longer good or belong to best practices to block indexing “/wp-content/plugins”.

    I’m not an expert but have a feeling these pages should not be appearing in search results or showing up as errors in Google search console.
    Am I wrong in both cases?

    If not, how can this be prevented?

    What should a basic WordPress robots.txt look like that does not lead to any errors in Google console look like or can one ignore some of them and if so, which ones?

    Actually I have had this plugin for the last 3 years and never had these particular errors before, though ever since I changed to https from http about 2, 3 months ago, I have had diverse errors, which have been dealt with by adjusting robots.txt.

    Currently, my robots.txt looks like this:

    User-agent: *
Disallow: /suggest/?*
Disallow: /readme.html
Disallow: /logout*
Disallow: /login*
Disallow: /register*
Disallow: /lostpassword*
Disallow: /resetpass*
Disallow: /trackback/
Disallow: /xmlrpc.php
Disallow: /*?*
Allow: /*.js
Allow: /*.css

    The last 3 lines have been added recently to prevent a flurry of some funny errors related to such urls “site.com/directory/listing/allafrica-com-djibouti?tab=related&category=0&is_mile=0&directory_radius=0&view=map&is_drag=1&center=#sabai-inline-content-related” mostly ending in 500 i.s.errors (the only uselful part needing indexing per directory listing being before the “?”). These particular ones are generated by a particular web directory plugin.

    Before that, lines 4 – 8 were added to prevent lots of warnings about links, especially those looking like “/logout/?_wpnonce=3959b740db”.

    I would appreciate some clear answers.

    Best regards

    • This topic was modified 7 years, 11 months ago by xprt007.
    • This topic was modified 7 years, 11 months ago by xprt007.
    • This topic was modified 7 years, 11 months ago by xprt007.

    The page I need help with: [log in to see the link]

Viewing 4 replies - 1 through 4 (of 4 total)
  • Jacob Peattie

    (@jakept)

    7 years, 11 months ago

    What should a basic WordPress robots.txt look like that does not lead to any errors in Google console look like or can one ignore some of them and if so, which ones?

    You should not see any errors at all, even without a robots.txt file. You should not be required to add anything to a robots file to prevent them.

    The only reason Google would be able to crawl these files is if there were links to them on the public side of your site somewhere, or you manually told Google about them.

    The reason they’re returning 500 errors is because those files are PHP scripts that are not designed to be run directly. They expect other parts of WordPress to already be loaded. So the real issue here is figuring out why they are being crawled at all.

    The first possibility that came to mind is that you have not disabled directory browsing on your server. So I tried to visit https://www.habarisearch.com/wp-content/plugins/recipepress-reloaded/, which confirmed that this is the issue. You should not be able to see those links to those files. This is why Google is able to crawl them. Ideally the plugin would have a blank index.php file in its folders, that would prevent this, but the real solution is to disable directory browsing on your server. This guide tells you how to do this in your htaccess file, but I’d suggest asking your host for help disabling it across your entire hosting account.

    • This reply was modified 7 years, 11 months ago by Jacob Peattie.
    • This reply was modified 7 years, 11 months ago by Jacob Peattie.
    Thread Starter xprt007

    (@xprt007)

    7 years, 11 months ago

    Hi there
    Thank you for the quick response and help.
    I was able to disable directory browsing via Cpanel. Now That link above leads to a page not found error, which I hope is expected.

    If I may ask, though, with the above robots.txt “/logout/?_wpnonce=3959b740db” is blocked. If I remove all but the first 3 lines in robots.txt, it’s not blocked. I have had Google console give I thnk warnings in the past of variations of “/logout/?_wpnonce=3959b740db”, and some links affected by /logout & so on I think.

    With the 3 lines “/logout/?_wpnonce=3959b740db” is not blocked in Google robots.txt tester. Is that a cause for worry, especially considering your first statement above, of normally there being no errors at all?

    Regards

    Jacob Peattie

    (@jakept)

    7 years, 11 months ago

    It’s probably normal to have a logout link somewhere on a page, so it’s not surprising that Google can find it, but ideally that link should probably have a nofollow on it, or the destination have a noindex or something. How is that link generated? Is it by a theme or plugin? And what are the specific warnings that you see?

    I should also clarify that when I said “You should not see any errors at all” I was just referring to the 500 response code error. That indicates a code error and should not exist on any crawl-able link. There might still be Google-specific warnings in the search console, but whether or not they’re a problem depends on the specific warning.

    • This reply was modified 7 years, 11 months ago by Jacob Peattie.
    Thread Starter xprt007

    (@xprt007)

    7 years, 11 months ago

    Hi

    I probably did that the other day by following instructions by Yoast CEO plugin folks who have the option of turning off “Allow search engines to show this page in search results” & “Should search engines follow links on this page?”.

    I set both for those specific pages to “No”.
    I had deleted all errors messages that day but will watch if anything comes up again.

    Thanks for your help.

    • This reply was modified 7 years, 11 months ago by xprt007.
Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘Crawl errors related to plugins?’ is closed to new replies.