Better Robots.txt – Index, Rank & SEO booster by Pagup

Description

Better Robots.txt creates a virtual robots.txt and helps you boosting your website indexing and ranking with just a few clicks.

You can also identify which search engines are allowed to crawl (or not) your website and specify clear instructions about what they are allowed to do or not, specify a crawl-delay (to protect your hosting server) and gives you a full control over the Robots.txt file content via custom setting box.

About the Pro version:

Better Robots.txt allows you to add automatically your sitemap (generated from YOAST SEO) to your robots.txt or you can specify a custom sitemap URL. Also, Better Robots.txt features a tool allowing you to block most popular bad bots from scraping illegally your data and content. And Better Robots.txt helps you block most SEO crawlers to keep your backlinks undetectable (by the competition)

Screenshots

  • Better Robots.txt Settings Page
  • Better Robots.txt Settings Page
  • Robots.txt file output

Installation

Installing manually

  1. Unzip all files to the /wp-content/plugins/better-robots-txt directory
  2. Log into WordPress admin and activate the ‘Better Robots.txt’ plugin through the ‘Plugins’ menu
  3. Go to “Settings > Better Robots.txt” in the left-hand menu to start work on robots.txt file.

FAQ

Installation Instructions

Installing manually

  1. Unzip all files to the /wp-content/plugins/better-robots-txt directory
  2. Log into WordPress admin and activate the ‘Better Robots.txt’ plugin through the ‘Plugins’ menu
  3. Go to “Settings > Better Robots.txt” in the left-hand menu to start work on robots.txt file.
Better Robots.txt plugin is enabled but why can’t i see any changes in robots.txt file?

Better Robots.txt creates a virtual robots.txt file. Please make sure that your permalinks are enabled from Settings > Permalinks. If permalinks are working then make sure that there is no physical robots.txt file on your server. Since it can’t write over physical file, so you must connect to FTP and rename or delete robots.txt from your domain root directory. It usually in /public_html/ folder on cPanel hostings. If you can’t find your domain root directory, please ask your hosting provider for help. If issue persists after taking these measures, please post it in support section or send a message to support@better-robots.com

Will there be any conflict with robots.txt which i’m already using?

If you have a pshysical robots.txt on your web hosting server, then this plugin will not work. As mentioned, it creates a virtual robots.txt file. Please follow steps in above answer if you want to use robots.txt file with this plugin.

How to add sitemap in robots.txt?

This feature is allowed in Better Robots.txt Pro version, which automatically add sitemap in robots.txt file. It detects sitemap from Yoast SEO plugin. In-case you’re using a different sitemap plugin or a manually generated sitemap then you can simply add sitemap URL in sitemap input field. If Yoast XML sitemaps are also enabled then you need to disable it first by simply going to Yoast General Settings > Features and disable XML Sitemaps feature.

Why should I optimise the robots.txt?

Why not? Considering that the robots.txt is the very first file read when your website is loaded by a browser, why not enable crawlers to continuously index your content? The simple fact of adding your Sitemap in the Robots.txt is simply common sense. Why? Did you list your website on Google Search Console, did your webmaster do it? How to tell the crawlers that you have new content available for indexation on your website? If you want this content to be found on search engines (Google, Bing, …), you have to have it indexed. That’s exacly what this instruction (adding the sitemap) aim to. One last point. The main reason why this plugin exists is due to the fact that 95% of the time (based of thousands of SEO analysis), the robots.txt is either missing, empty or misued, simply because it is either misunderstood or forgotten. Imagine now if it was activated and fully functional.

How can this plugin boost my website ranking?

Actually, this plugin will increase your website indexation capacity which leads to improve your ranking on Google. How ? Well, the idea of creating this plugin was taken after making hundreds of SEO optimization on professional and corporative websites. As mentioned before, 95% of analysed websites did not have what we could call an “optimized” robots.txt file and, while we were optimizing these websites, we realized that simply modifying the content of this file was actually “unlocking” these websites (based on daily SEMrush analysises). As we were used to working in 2 steps (periods of time), starting with this simple modification was already generating a significant impact on Google Ranking, and this, even before we started deeply modifying either the content, the site arborescence or META Data. The more you help search engines at understanding your website, the better you help your capacity of getting better results on SERPs.

Reviews

Just learn to copy and paste!

This plugin is somewhat nice looking and allow you to change your robots file with a little interface, but why would you add another plugin to cluster your WP Admin? Just upload a robots.txt file at the root of your website and copy/paste your robots info in it.

Saved my SEO!

Better Robots.txt saved me. For an unknown reason one of my sites had a “hook” at ” Discourage search engines from indexing this site” under Setting/Reading in admin panel. I had no idea about this and fought for hours to find why Google and the othersa did not index my site. I found no Robots.txt showing why.

Then I discovered Better Robots.txt. It found a file, even though it could not locate it. This robots.txt file stopped indexing. But in cooperation with their support and changing my SEO solution to Yoast it now seems that my site will get higher ranking than ever before! Thank you Gautier Dorval and the team at Pagup in Canada!

NB! I have no affiliation with Pagup. They just have a great product and support

Easy to Use and Time Saver

I have been modifying robots.txt files manually for years but this plugin is a real time saver. Better Robots.txt create robots.txt file with simple clicks even new user can easily use it.

Read all 3 reviews

Contributors & Developers

“Better Robots.txt – Index, Rank & SEO booster by Pagup” is open source software. The following people have contributed to this plugin.

Contributors

“Better Robots.txt – Index, Rank & SEO booster by Pagup” has been translated into 1 locale. Thank you to the translators for their contributions.

Translate “Better Robots.txt – Index, Rank & SEO booster by Pagup” into your language.

Interested in development?

Browse the code, check out the SVN repository, or subscribe to the development log by RSS.

Changelog

1.0.0

  • Initial release.

1.0.1

  • fixed plugin directory url issue
  • some text improvements

1.0.2

  • fixed some minor issues with styling
  • improved text and translation

1.1.0

  • added some major improvements
  • allow/off option changed with allow/disallow/off
  • improved overall text and french translation

1.1.1

  • fixed a bug and improved code