WordPress.org

Ready to get started?Download WordPress

Forums

Is my Robots.txt file correct? (5 posts)

  1. TheGift73
    Member
    Posted 1 year ago #

    Hi,

    I have recently updated my Robots.txt file and was wondering if I am blocking items that I should. Also, the way I currently have it, is it doing any harm to my site ranking/indexing wise?

    Regards,

    Richard

    My site is: http://techfleece.com/

    My robots.txt file reads as:

    User-agent: *
    
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content
    
    User-agent: Mediapartners-Google
    Allow: /
    
    User-agent: Adsbot-Google
    Allow: /
    
    User-agent: Googlebot-Image
    Allow: /
    
    User-agent: Googlebot-Mobile
    Allow: /
    
    User-Agent: bingbot
    Allow: /
  2. Kevin Ashbridge
    Member
    Posted 1 year ago #

    That looks like a good WP robots.txt file to me.

    Google will rank you on the quality, freshness and relevancy of your content, plus the social proof around it. That's the important stuff to concentrate on.

    -- Kevin.

  3. TheGift73
    Member
    Posted 1 year ago #

    Hi Kevin,

    That's good to know. It was mainly as I seem to rank well for Google queries (average 1,400+ a day) but poorly on Bing (6 or 7 hits a day) I was just worried that Bing was being blocked from certain things that it needs in order to index correctly.

    Glad it looks OK to you tho.

    Thanks Kevin.

    --Richard

  4. Kevin Ashbridge
    Member
    Posted 1 year ago #

    Probably fewer of your users use Bing (!)

    The thing to check is if you both rank reasonably equally within both engines for the same keywords - such as 'both on page 2'. Then at least you know your website is returned consistently in the search results of either engine for your search terms.

  5. TheGift73
    Member
    Posted 1 year ago #

    Hi Kevin,

    I just did a quick check on both search engines for the exact same terms for my most viewed articles. The results are below. The only other thing that I can think of is that I don't use Tags or Categories (very rarely) on any of my posts. Maybe Bing's algorithm requires these to be present in order to rank/index correctly?

    Search Term: "Not enough server storage is available to process this command"

    Google - Page 1 at the top
    Bing - Checked up to page 10 and still not showing

    Search Term: "Take Ownership hard drive"

    Google - Page 1 at the top
    Bing - Page 2

    Search Term: "Split MKV files"

    Google - Page 1 at the top
    Bing - Page 1 in the middle

    Search Term: "Convert RAW Files To JPEG"

    Google - Page 1 at the top (no.3)
    Bing - Checked up to page 10 and still not showing

    It's interesting to see how the various engines check though. There are many articles that rank for a specific term that actually lack the particular relevancy.

    Richard.

Topic Closed

This topic has been closed to new replies.

About this Topic