WordPress.org

1. thedevnull
   Member
   Posted 1 year ago #

   Hello Yall,

   Anyone know why there is no robots.txt file included in WP? Arguably there are a few directories any admin wouldn't want showing up for security and SEO reasons. A few examples are:
   Disallow: /wp-
   Disallow: /feed/
   Disallow: /trackback/
   Disallow: /rss/
   Disallow: /*?*

   I have found that various engines will index places such as wp-* that one might not want indexed. =(

   Any idea on why there is no default robots.txt?

2. iridiax
   Member
   Posted 1 year ago #

   A robots.txt goes in the root directory, while WordPress may be installed in a subdirectory. Also, a site may already have a robots.txt with other non-WordPress things in it.

   WordPress will create a basic robots.txt if it's installed in the root directory, there is no existing robots.txt, and your privacy settings are set to private.

   BTW, I'd recommend not blocking your feed if you want to be indexed quickly by Google. Google is smart enough now not to consider a feed to be duplicate content, and if your feed is being indexed instead of your regular posts/pages, then you have something else wrong with your site.

3. thedevnull
   Member
   Posted 1 year ago #

   Hello Iridiax,

   Assumptions are a big problem here. We cannot assume where one is going to install WP or how it will be configured. Not having a default WP robots.txt file allow many files that should NOT be indexed to show up in most engines.

   FYI,that was just an example and I don't have that exact config. =P The "privacy" feature you are speaking doesn't address the issue that I am attempting to illuminate. A default WP robots.txt would give a user a general guide on what to NOT allow indexing and now no such info exists.

4. Gouri
   Member
   Posted 2 months ago #

   Is robots.txt file necessary? While setting up XML sitemap generator plugin, I had noticed that a virtual robots.txt is generated by WordPress & a real robots.txt file should NOT exist in the blog directory. I also felt that after installing real robots.txt file, my blog started performing poorly in google search.

5. Kawauso
   Member
   Posted 2 months ago #

   Gouri: The sitemap plugin will be using the virtual robots.txt to specify the sitemap location, rather than using it to restrict what's indexable, so in some cases a real one is necessary.

6. Gouri
   Member
   Posted 2 months ago #

   Thanks, Kawauso.

Topic Closed

This topic has been closed to new replies.