Title: Crawl errors related to plugins?
Last modified: July 2, 2018

---

# Crawl errors related to plugins?

 *  [xprt007](https://wordpress.org/support/users/xprt007/)
 * (@xprt007)
 * [7 years, 11 months ago](https://wordpress.org/support/topic/crawl-errors-related-to-plugins/)
 * Hi there
 * I have been having the following crawling errors in Google console, producing
   response code 500:
 * wp-content/plugins/recipepress-reloaded/lib/tgm/class-tgm-plugin-activation.php
   
   wp-content/plugins/recipepress-reloaded/views/metabox_ingredients.php wp-content/
   plugins/recipepress-reloaded/views/metabox_instructions.php wp-content/plugins/
   recipepress-reloaded/views/metabox_notes.php wp-content/plugins/recipepress-reloaded/
   views/rpr_recipe_public.php wp-content/plugins/recipepress-reloaded/widgets/rpr_widget_tag_cloud.
   php wp-content/plugins/recipepress-reloaded/views/admin.php wp-content/plugins/
   recipepress-reloaded/views/metabox_details.php wp-content/plugins/recipepress-
   reloaded/views/settings.php
 * I have read in several posts online regarding robots.txt & co, that it’s no longer
   good or belong to best practices to block indexing “/wp-content/plugins”.
 * I’m not an expert but have a feeling these pages should not be appearing in search
   results or showing up as errors in Google search console.
    Am I wrong in both
   cases?
 * If not, how can this be prevented?
 * **What should a basic WordPress robots.txt look like that does not lead to any
   errors in Google console look like or can one ignore some of them and if so, 
   which ones?**
 * Actually I have had this plugin for the last 3 years and never had these particular
   errors before, though ever since I changed to https from http about 2, 3 months
   ago, I have had diverse errors, which have been dealt with by adjusting robots.
   txt.
 * Currently, my robots.txt looks like this:
 *     ```
       User-agent: *
       Disallow: /suggest/?*
       Disallow: /readme.html
       Disallow: /logout*
       Disallow: /login*
       Disallow: /register*
       Disallow: /lostpassword*
       Disallow: /resetpass*
       Disallow: /trackback/
       Disallow: /xmlrpc.php
       Disallow: /*?*
       Allow: /*.js
       Allow: /*.css
       ```
   
 * The last 3 lines have been added recently to prevent a flurry of some funny errors
   related to such urls “site.com/directory/listing/allafrica-com-djibouti?tab=related&
   category=0&is_mile=0&directory_radius=0&view=map&is_drag=1&center=#sabai-inline-
   content-related” mostly ending in 500 i.s.errors (the only uselful part needing
   indexing per directory listing being before the “?”). These particular ones are
   generated by a particular web directory plugin.
 * Before that, lines 4 – 8 were added to prevent lots of **warnings** about links,
   especially those looking like “/logout/?_wpnonce=3959b740db”.
 * I would appreciate some clear answers.
 * Best regards
    -  This topic was modified 7 years, 11 months ago by [xprt007](https://wordpress.org/support/users/xprt007/).
    -  This topic was modified 7 years, 11 months ago by [xprt007](https://wordpress.org/support/users/xprt007/).
    -  This topic was modified 7 years, 11 months ago by [xprt007](https://wordpress.org/support/users/xprt007/).
 * The page I need help with: _[[log in](https://login.wordpress.org/?redirect_to=https%3A%2F%2Fwordpress.org%2Fsupport%2Ftopic%2Fcrawl-errors-related-to-plugins%2F%3Foutput_format%3Dmd&locale=en_US)
   to see the link]_

Viewing 4 replies - 1 through 4 (of 4 total)

 *  [Jacob Peattie](https://wordpress.org/support/users/jakept/)
 * (@jakept)
 * [7 years, 11 months ago](https://wordpress.org/support/topic/crawl-errors-related-to-plugins/#post-10454364)
 * > What should a basic WordPress robots.txt look like that does not lead to any
   > errors in Google console look like or can one ignore some of them and if so,
   > which ones?
 * You should not see any errors at all, even without a robots.txt file. You should
   not be required to add anything to a robots file to prevent them.
 * The only reason Google would be _able_ to crawl these files is if there were 
   links to them on the public side of your site somewhere, or you manually told
   Google about them.
 * The reason they’re returning 500 errors is because those files are PHP scripts
   that are not designed to be run directly. They expect other parts of WordPress
   to already be loaded. So the real issue here is figuring out why they are being
   crawled _at all_.
 * The first possibility that came to mind is that you have not disabled directory
   browsing on your server. So I tried to visit [https://www.habarisearch.com/wp-content/plugins/recipepress-reloaded/](https://www.habarisearch.com/wp-content/plugins/recipepress-reloaded/),
   which confirmed that this is the issue. You should not be able to see those links
   to those files. This is why Google is able to crawl them. Ideally the plugin 
   would have a blank index.php file in its folders, that would prevent this, but
   the real solution is to disable directory browsing on your server. [This guide](https://www.thesitewizard.com/apache/prevent-directory-listing-htaccess.shtml)
   tells you how to do this in your htaccess file, but I’d suggest asking your host
   for help disabling it across your entire hosting account.
    -  This reply was modified 7 years, 11 months ago by [Jacob Peattie](https://wordpress.org/support/users/jakept/).
    -  This reply was modified 7 years, 11 months ago by [Jacob Peattie](https://wordpress.org/support/users/jakept/).
 *  Thread Starter [xprt007](https://wordpress.org/support/users/xprt007/)
 * (@xprt007)
 * [7 years, 11 months ago](https://wordpress.org/support/topic/crawl-errors-related-to-plugins/#post-10454803)
 * Hi there
    Thank you for the quick response and help. I was able to disable directory
   browsing via Cpanel. Now That link above leads to a page not found error, which
   I hope is expected.
 * If I may ask, though, with the above robots.txt “/logout/?_wpnonce=3959b740db”
   is blocked. If I remove all but the first 3 lines in robots.txt, it’s not blocked.
   I have had Google console give I thnk warnings in the past of variations of “/
   logout/?_wpnonce=3959b740db”, and some links affected by /logout & so on I think.
 * With the 3 lines “/logout/?_wpnonce=3959b740db” is not blocked in Google robots.
   txt tester. Is that a cause for worry, especially considering your first statement
   above, of normally there being no errors at all?
 * Regards
 *  [Jacob Peattie](https://wordpress.org/support/users/jakept/)
 * (@jakept)
 * [7 years, 11 months ago](https://wordpress.org/support/topic/crawl-errors-related-to-plugins/#post-10454914)
 * It’s probably normal to have a logout link somewhere on a page, so it’s not surprising
   that Google can find it, but ideally that link should probably have a `nofollow`
   on it, or the destination have a `noindex` or something. How is that link generated?
   Is it by a theme or plugin? And what are the specific warnings that you see?
 * I should also clarify that when I said “You should not see any errors at all”
   I was just referring to the 500 response code error. That indicates a code error
   and should not exist on any crawl-able link. There might still be Google-specific
   warnings in the search console, but whether or not they’re a problem depends 
   on the specific warning.
    -  This reply was modified 7 years, 11 months ago by [Jacob Peattie](https://wordpress.org/support/users/jakept/).
 *  Thread Starter [xprt007](https://wordpress.org/support/users/xprt007/)
 * (@xprt007)
 * [7 years, 11 months ago](https://wordpress.org/support/topic/crawl-errors-related-to-plugins/#post-10455434)
 * Hi
 * I probably did that the other day by following instructions by Yoast CEO plugin
   folks who have the option of turning off “Allow search engines to show this page
   in search results” & “Should search engines follow links on this page?”.
 * I set both for those specific pages to “No”.
    I had deleted all errors messages
   that day but will watch if anything comes up again.
 * Thanks for your help.
    -  This reply was modified 7 years, 11 months ago by [xprt007](https://wordpress.org/support/users/xprt007/).

Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘Crawl errors related to plugins?’ is closed to new replies.

 * In: [Fixing WordPress](https://wordpress.org/support/forum/how-to-and-troubleshooting/)
 * 4 replies
 * 2 participants
 * Last reply from: [xprt007](https://wordpress.org/support/users/xprt007/)
 * Last activity: [7 years, 11 months ago](https://wordpress.org/support/topic/crawl-errors-related-to-plugins/#post-10455434)
 * Status: not resolved

## Topics

### Topics with no replies

### Non-support topics

### Resolved topics

### Unresolved topics

### All topics