WordPress.org

Support

Support » How-To and Troubleshooting » Is my Robots.txt file correct?

Is my Robots.txt file correct?

  • Hi,

    I have recently updated my Robots.txt file and was wondering if I am blocking items that I should. Also, the way I currently have it, is it doing any harm to my site ranking/indexing wise?

    Regards,

    Richard

    My site is: http://techfleece.com/

    My robots.txt file reads as:

    User-agent: *
    
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content
    
    User-agent: Mediapartners-Google
    Allow: /
    
    User-agent: Adsbot-Google
    Allow: /
    
    User-agent: Googlebot-Image
    Allow: /
    
    User-agent: Googlebot-Mobile
    Allow: /
    
    User-Agent: bingbot
    Allow: /
Viewing 4 replies - 1 through 4 (of 4 total)
  • That looks like a good WP robots.txt file to me.

    Google will rank you on the quality, freshness and relevancy of your content, plus the social proof around it. That’s the important stuff to concentrate on.

    — Kevin.

    Hi Kevin,

    That’s good to know. It was mainly as I seem to rank well for Google queries (average 1,400+ a day) but poorly on Bing (6 or 7 hits a day) I was just worried that Bing was being blocked from certain things that it needs in order to index correctly.

    Glad it looks OK to you tho.

    Thanks Kevin.

    –Richard

    Probably fewer of your users use Bing (!)

    The thing to check is if you both rank reasonably equally within both engines for the same keywords – such as ‘both on page 2’. Then at least you know your website is returned consistently in the search results of either engine for your search terms.

    Hi Kevin,

    I just did a quick check on both search engines for the exact same terms for my most viewed articles. The results are below. The only other thing that I can think of is that I don’t use Tags or Categories (very rarely) on any of my posts. Maybe Bing’s algorithm requires these to be present in order to rank/index correctly?

    Search Term: “Not enough server storage is available to process this command”

    Google – Page 1 at the top
    Bing – Checked up to page 10 and still not showing

    Search Term: “Take Ownership hard drive”

    Google – Page 1 at the top
    Bing – Page 2

    Search Term: “Split MKV files”

    Google – Page 1 at the top
    Bing – Page 1 in the middle

    Search Term: “Convert RAW Files To JPEG”

    Google – Page 1 at the top (no.3)
    Bing – Checked up to page 10 and still not showing

    It’s interesting to see how the various engines check though. There are many articles that rank for a specific term that actually lack the particular relevancy.

    Richard.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Is my Robots.txt file correct?’ is closed to new replies.