Major Google Search Console errors from Robots.txt

  • JohnnyScience

    (@johnnyscience)


    6 years, 9 months ago

    So I’ve been dealing with this problem for a long time and I don’t know why I haven’t really tried addressing it until now.

    Honestly I don’t have a clue to fix any of it and it just seemed crippling.

    But I’ve continued to get new Google Search Console error emails

    I’m not sure how or why it happened, but the major errors are apparently resulting from problems with my robots.txt file

    I don’t know exactly what the problems are, but you can review them by running the google search test on my url here:

    https://search.google.com/test/mobile-friendly?utm_source=mft&utm_medium=redirect&utm_campaign=mft-redirect

    I never edited this file, so I don’t know why it’s so corrupted and giving me all of these errors.

    Can anyone help me clean this file up and get rid of these errors?

    Thank you!

    • This topic was modified 6 years, 9 months ago by JohnnyScience.

    The page I need help with: [log in to see the link]

Viewing 8 replies - 1 through 8 (of 8 total)
  • Moderator Steven Stern (sterndata)

    (@sterndata)

    Volunteer Forum Moderator

    6 years, 9 months ago

    That page says nothing about your robots.txt

    Thread Starter JohnnyScience

    (@johnnyscience)

    6 years, 9 months ago

    Sorry, where/how can I post my robots.txt file for review to help figure out exactly what’s going on?

    I was told that it needs to be “configured” and need someone who understands SEO

    • This reply was modified 6 years, 9 months ago by JohnnyScience.
    Moderator Steven Stern (sterndata)

    (@sterndata)

    Volunteer Forum Moderator

    6 years, 9 months ago

    Paste it here.

    Thread Starter JohnnyScience

    (@johnnyscience)

    6 years, 8 months ago

    According to Yoast SEO, I don’t even have a robots.txt

    Is this normal?

    The Yoast plugin gives me the option to create one, but I’m not quite sure what info it needs.

    Also, if I go to http://www.ohiocannabis.com/robots.txt, I get this:

    Sitemap: https://www.ohiocannabis.com/sitemap.xml
    Sitemap: https://www.ohiocannabis.com/news-sitemap.xml
    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    What should I do next, have the Yoast plugin create a robots.txt file for me?

    How/what should I structure within it so Google finally crawls my website properly and I don’t continue to have all of these Google Search Console errors?

    Thank you!

    Thread Starter JohnnyScience

    (@johnnyscience)

    6 years, 8 months ago

    This is currently my new robots.txt file I created with Yoast SEO:

    ​​​User-agent: *
    Allow: /wp-content/uploads/
    Disallow: /wp-content/plugins/
    Disallow: /wp-admin/
    Disallow: /readme.html
    Allow: /wp-admin/admin-ajax.php

    Sitemap: https://www.ohiocannabis.com/sitemap.xml
    Sitemap: https://www.ohiocannabis.com/news-sitemap.xml

    It seems all of my crawl issues are related to images for the most part.

    But for some reason when I click on “See Live Robots.txt file” in the Google Robots.txt tester, it is still populating this old robots.txt file:

    Sitemap: https://www.ohiocannabis.com/sitemap.xml
    Sitemap: https://www.ohiocannabis.com/news-sitemap.xml
    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    I’ve verified the actual robots.txt file in FTP and it shows the updated robots.txt file that Yoast SEO has created.

    So why is this “See Live Robots.txt File” option still showing the old, default file?

    Shouldn’t the Allow: /wp-content/uploads/ clear up all of these image crawl issues now that I’ve added that specific Allow to the robots.txt file?

    Also, is it necessary to have both of those Sitemaps? Is the news-sitemap.xml really needed? It was auto-generated by Yoast SEO when I had it create the robots.txt file.

    I hope someone can help me get this fixed already, I’ve been getting destroyed with this terrible Indexing for over a year. I even get errors that my Home page isn’t Indexed!

    Finally, when it comes to “Properties” in Google Search Console, I was originally told to add the following 4 Properties by someone:

    https://www.ohiocannabis.comm
    http://www.ohiocannabis.com
    https://ohiocannabis.com
    http://ohiocannabis.com

    I’ve come to realize only the https://www.ohiocannabis.com had any real stats, the other 3 Properties literally had 4 crawls on them.

    I did notice that my robots.txt file was only updated on the https://www.ohiocannabis.com URL, for some reason the other 3 kept the old/default robots.txt file

    Could perhaps this be part of the problems I’ve been having?

    I just deleted the other 3 as I didn’t think they were doing anything based on their stats and clearly seemed to be negatively affecting Indexing with the old/default robots.txt file

    Why wouldn’t these 3 other Properties have their robots.txt files updated when I had Yoast SEO create a new one for me?

    Should I add them back?

    Sorry this is so long, but I’m really in a bad place with SEO & Indexing and it’s really been devastating to my business this last year by having very low traffic.

    Thank you.

    [Moderator note: Please, No bumping.

    Thread Starter JohnnyScience

    (@johnnyscience)

    6 years, 8 months ago

    I tried posting all of this over on the Google Search Console support forums, however they deleted my thread because of the word cannabis in it. So I won’t be able to get any help from them at all it seems. Hopefully someone here can help me get this figured out, it’s a total mess.

    Even my main Home Page hasn’t been Indexing this entire time, which according to google means it’s never showing up for any searches – which is obviously a major problem since I have a lot of pertinent info on my Home Page.

    I would like to update that my new Yoast SEO robots.txt seems to be populating properly.

    However, the Allow: /wp-content/uploads/ doesn’t seem to have helped google crawl the robots.txt, here’s a screenshot:

    Which I’m really confused how the ability of the images to be crawled/featured/populate on my wordpress site has been messed up since install?

    And even with adding this new Allow: /wp-content/uploads

    Here is another screenshot showing the robots.txt is still blocked:

    I’m not sure if this robots.txts affects the mobile-friendly test coming back with a bunch of resources errors or not still or is this a separate problem still?

    • This reply was modified 6 years, 8 months ago by JohnnyScience.
    Thread Starter JohnnyScience

    (@johnnyscience)

    6 years, 7 months ago

    I’ve gotten most of these cleared up one way or another.

    Does anyone else have any input on what else I could be doing to fix all of the remaining errors?

    Thank you.

    Thread Starter JohnnyScience

    (@johnnyscience)

    6 years, 7 months ago

    I contacted Hostgator about the speed of my website and they said they have taken a look at the server and I do not see any issues on the server end.

    But they did say this was coming up as a problem:

    I can see from this gtmetrix test that your site makes over 278 requests mostly from images. You would need to work on optimize the content so that it required less request and data to perform a single site request. 

https://gtmetrix.com/reports/ohiocannabis.com/IsDt1bxv

    So could this be the reason my home page is also loading so slow?

    Any ideas on how to clear up/reduce all of those image requests?

    I also contacted Sitelock & the had this to say:

    During our clean of your website, we reviewed the search engine results to see if there was spam content present, checked your site against various blacklists to make sure your site wasn't being blacklisted, checked your sites database, and performed a thorough review of your sites files. We can confirm that there is no spam content in the search results and that your site is not on any blacklist, additionally, we found your database and files to be free of malicious content. We then also tested your site to make sure everything was functioning properly. We can also confirm that from our end the site is loading slowly. The next steps after ruling out malware would be to analyze waterfalls for the loading of the site and attempt to narrow down which part of the site is causing delayed load times. Typically the best approach is to attempt to disable plugins or change the theme and compare and contrast the response of the page. Ideally, if it can be narrowed down to a few individual plugins

    Thank you.

Viewing 8 replies - 1 through 8 (of 8 total)

The topic ‘Major Google Search Console errors from Robots.txt’ is closed to new replies.