Support » Fixing WordPress » Robots,text file help

  • I have a problem with google and other search engines indexing all images in wordpress media as a post with empty content.

    I have created this robots.text fiel to avoid this and other problems but i am not sure if it is correct

    Could somoen tell if the file is ok, should i add something else or take off something from the file?

    User-agent: *
    User-agent: Googlebot-Image
    Disallow: /
    User-agent: msnbot-media
    Disallow: /
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/
    Disallow: /search
    Disallow: /account
    Disallow: /login`

Viewing 1 replies (of 1 total)
  • First off, you stopped Google Image bot & MSNbot from indexing your site at all.

    User-agent: Googlebot-Image
    Disallow: /
    User-agent: msnbot-media
    Disallow: /

    The above states to Disallow everything.

    Disallowing in ‘robots.txt’ is fine if nothing has already been indexed. Presuming images already have been, you’ll want to remove any image blocking in the robots.txt file because if the bots cannot access images, they cannot remove them from their index. Sounds odd, I know, but that’s the way it works.

    The only way to do what you want, prevent further indexing and remove the images already indexed, is with the following.

    In robots.txt file

    User-agent: *
    Disallow: /wp-admin/

    Copy & Paste this in your .htacces file, somewhere above the WordPress code.

    <FilesMatch “\.(gif|jpg|png|webp)$”>
    Header always set x-robots-tag: noindex
    </FilesMatch>

    Code for the .htaccess file will tell the Search Bots you don’t want those image types indexed and if they are already indexed, then de-index them.

    Now go to Google and type in: image site:yourdomain.com

    On the Results page near page top you’ll see: All – Images – Videos – etc.

    Click “Images” to see the ones already indexed. As it will take a long time to de-index all your images (we’re talking months), check once a month to see how many there are.

    Once there are none, add this to your ‘robots.txt’ file.

    Disallow: /wp-content/
    Disallow: /wp-includes/

    You can then remove the code from your .htaccess file.

    Be aware that the changes may not work out the way you want as far what Search Engines show for anything image related.

    If not happy with the results, remove the code from the .htaccess file, don’t Disallow in the ‘robots.txt’ file and, after a few months, your images will start to appear again.

Viewing 1 replies (of 1 total)
  • The topic ‘Robots,text file help’ is closed to new replies.