WordPress.org

Ready to get started?Download WordPress

Forums

[resolved] WordPress update Google banning my site due to robot file (13 posts)

  1. Carri4318
    Member
    Posted 6 months ago #

    I have recently updated to WordPress 3.8.
    Yesterday someone contacted me to say they couldn't find our site:
    http://lincolnfilm.org.uk/ listed anywhere on the Google search engine. Since then I have been desperately trying to sort it out, and have narrowed the problem down to:
    Google couldn't crawl your site because we were unable to access your site's robots.txt file.

    I really, really don't know how to fix this. I'm pretty sure it's nothing I have done, particularly as I don't even know what a robot file is!

    Please HELP

  2. Robin W
    Member
    Posted 6 months ago #

    ok, start by reading this :

    http://www.robotstxt.org/robotstxt.html

    Then if you enter your site and look at that file ie go to

    http://lincolnfilm.org.uk/robots.txt

    you see that it is all set to disallow.

    The I'd check

    Dashboard>settings>reading and see what setting you have for search engine visibility. If set, the unset it. if unset then set and save and inset and save and see if that changes the robots file.

    If it doesn't, the edit your robots.txt directly using FTP and an editor

    change the file to read

    User-agent: *
    Crawl-delay: 2

    and that's all

  3. Carri4318
    Member
    Posted 6 months ago #

    Thanks so much for the help. I managed to follow you to the point of:
    If set, the unset it. if unset then set and save and inset and save and see if that changes the robots file.

    If it doesn't, the edit your robots.txt directly using FTP and an editor

    change the file to read
    User-agent: *
    Crawl-delay: 2
    and that's all
    But got lost with your instructions for the if set etc. The box is not ticked.
    I really would need a lot more instructions to edit the robots.txt, as I don't know what FTP is or how to edit it.
    I have desperately tried to get on a website design course, but the only ones here are for 2 hours a week for 4 weeks, which actually can't learn anything in. So basically everything I do know is home taught (the hard way!)
    So ANY help is always gratefully received
    thanks
    carrie

  4. Robin W
    Member
    Posted 6 months ago #

    Ok, so if the box is not ticked, then tick it and save.

    Then go back in and untick it and save.

    With luck that will alter the file.

    so go back into
    http://lincolnfilm.org.uk/robots.txt

    and see if it has changed.

    If yes, and the disallows have gone then you should be fixed.

    If not, then come back and I'll help further

  5. Carri4318
    Member
    Posted 6 months ago #

    Hi
    tried it, but it didnt work unfortunately.
    I just dont understand why Google have all of a sudden got such a problem with the robots. Its not as thought I changed anything!
    Anymore ideas would be gratefully received
    thanks
    carrie

  6. Robin W
    Member
    Posted 6 months ago #

    as to why, I can't answer - something has changed it.

    ok, so you need to create a new robots.txt and upload it to overwrite your existing one.

    To do this you'll need to

    1. create the file
    2. use FTP to upload

    1. Use notepad (you'll find it in accessories) and just copy this into it

    User-agent: *
    Crawl-delay: 2

    Save this file as robots.txt to your documents area (or any other area where you can find it again

    That's the easy part.

    2. To use FTP you need
    a) An FTP programme to use to upload.
    b) the access details - the host, username and password of your FTP account

    a)
    It may be that you host provider gives you the programme to allow FTP - go into your host account and see if it is there or discuss with you host provider. If so then you can just upload it - you'll find the existing robots.txt in the root of you website files - just look around for it, and then upload and overwrite it with the new one you've created.

    If not you'll need to download one. One of the most used is filezilla client.

    Watch this video

    http://www.youtube.com/watch?v=Wtqq1Mn1ltA

    start by going as far as 2.30 mins (that'll get you the install)

    Then you'll need you host details and you'll use this instead of the "local" he uses in the rest of the video.

    b) access details
    Your web provider account should have these details - basically you're looking for "FTP account" or similar. If you can't find this, talk to your host provider - they will know what to tell you, just ask for your FTP account host, username and password

    Armed with this you can watch the remainder of the video, and learn how to move a file to your server. You'll find the existing robots.txt in the root of you website files - just look around for it, and then upload and overwrite it with the new one you've created.

    All the above looks daunting - but take it slowly and you'll find it's quite easy !

  7. Pioneer Valley Web Design
    Member
    Posted 6 months ago #

    A) If you have a robots.txt file in root, the virtual robots.txt file is not output.
    B) Google does not "Google banning my site due to robot file"
    C) Your current file is:

    User-agent: *
    Disallow: /forum/
    Disallow: /cgi-bin/
    Disallow: /2007/
    Disallow: /zxc/
    Disallow: /20052006season.html
    Disallow: /forum/viewforum
    Disallow: /tests/
    Disallow: /cinelatino.pdf
    Disallow: /Lincoln%20Film%20Society%201958-2006.pdf
    Disallow: /festival%20artwork
    Disallow: /filmlist5807.pdf
    Disallow: /pdfs/
    Disallow: /films/2006-07
    Disallow: /films/festival07
    Disallow: /images/
    Disallow: /a2z.html
    Disallow: /about.html
    Disallow: /archive
    Disallow: /archivebreadcrumb.html
    Disallow: /thankyou.html
    Disallow: /menu.html
    Disallow: /links.html
    Disallow: /festivalartwork.html
    Disallow: /committee.html
    Disallow: /jw/
    Disallow: /indeximages/
    Disallow: /media/
    
    Allow: /index.html

    Which is certainly not the WordPress created virtual file.

    1. The URL's in this list should exist (WP sites usually do not have .html files).
    2. If they do, check for errors at Google Webmaster Tools.

  8. Carri4318
    Member
    Posted 6 months ago #

    To Robin W
    I thought I had done everything right. But I did a fetch as google and it is still coming up with this message:
    Unreachable robots.txt

    Now I have a really big headache! Anymore ideas?

  9. Robin W
    Member
    Posted 6 months ago #

    Ok, well you've fixed the file, so at least that won't be in the way.

    What precisely do you mean by "But I did a fetch as google and it is still coming up with this message:"

  10. Carri4318
    Member
    Posted 6 months ago #

    Hi
    I went onto Goobgle Webmaster Tools > Crawl > Fetch as Google + Unreachable robots.txt

    Argh !
    Too much hair pulling now from me!

  11. Robin W
    Member
    Posted 6 months ago #

    Hmmm...clearly something is unhappy.

    Given that the old robots.txt file had a lot of non-wordpress file names - is this clean wordpress installation, or part of a bigger website or one that has had other stuff before?

    My next worry would be that the url is doing some sort of redirect - google doesn't like that, and will go away.

    In either event I suggest that you now delete the robots.txt file - this will then mean that wordpress supplies a virtual version.

  12. Robin W
    Member
    Posted 6 months ago #

    I'd also contact you host provider to see if they can help - you pay for your site so they should help you !

  13. Carri4318
    Member
    Posted 6 months ago #

    Hi
    I have repeatedly asked my host provider for help. They help a little and then that's it. So no more help forthcoming from them unfortunately.
    I took over the site from someone else a few years ago. I started using WordPress then (the other guy wasn't). But obviously somethings must have been transferred (I thought because I had changed to WordPress and built all the pages from scrap that nothing remained of the old site).
    How would I delete the robots.txt file? Using Filezilla?
    I just really don't understand why Google suddenly have aproblem!

Reply

You must log in to post.

About this Topic