Support » Plugins and Hacks » Robots.txt file problem using WordPress SEO by Yoast

Robots.txt file problem using WordPress SEO by Yoast

  • Hi,
    i didn’t knew i should post my problem here, so there it is:

    i have a problem with my robots.txt file.
    I was using Google XML Sitemaps and received messages in the webmaster tools that bots blocked my pages, i deleted the plugin, deleted the sitemaps.xml file from the root directory and switched to SEO by Yoast, i enabled the XML Sitemaps.
    I tested the new sitemap in the webmaster tools and it says:


    Url blocked by robots.txt.

    Sitemap contains urls which are blocked by robots.txt.


    Value: http://dateme.bg/category-sitemap.xml

    Mar 19, 2013
    Value: http://dateme.bg/page-sitemap.xml

    Mar 19, 2013
    Value: http://dateme.bg/post-sitemap.xml

    Here is my robots.txt file:
    # User-agent: *

    # Disallow: /wp-admin/

    # Disallow: /wp-includes/

    # Disallow: /wp-trackback

    # Disallow: /wp-feed

    # Disallow: /wp-comments

    # Disallow: /wp-content/plugins

    # Disallow: /wp-content/themes

    # Disallow: /wp-login.php

    # Disallow: /wp-register.php

    # Disallow: /feed

    # Disallow: /trackback

    # Disallow: /cgi-bin

    # Disallow: /comments

    # Disallow: *?s=

    I’m really sorry, but i have no idea what to do. I don’t have any nofollow links or metatags..

    I will appreciate any help!
    Thank you


Viewing 15 replies - 1 through 15 (of 24 total)
  • do you have a physical file by the name of robots.txt in your root directory?

    yes, i do!

    rename the robots.txt and the Yoast plugin will make on of its own. Or else, you could go to http://yoast.com/example-robots-txt-wordpress/ and make a new robot.txt

    User-Agent: *
    Disallow: /wp-content/plugins/

    this is the simplest of all.

    Hi again,
    i’m very sorry, but i’m a newbie. How should i name the robots.txt file? You say i should rename it? What name should i give it?

    its ok, we were all newbies when we started. just rename it to test.txt. if you really need help and wish to learn then do a teamviewer session with me. We could fix it together…

    Thank you for your understanding, i just did that.
    Should i wait before i check again at the webmaster tools? Because now it shows the same results, also when i test a sitemap, may i add the main sitemap, that Yoast makes – sitemap_index.xml or should i test them separately – for the post and the page?
    I love this plugin and highly appreciate your help! I’ve put a lot of effort in this website, please advise what to do further 🙂

    We all love Yoast 🙂
    you should have one sitemap which should be something like this

    http://www.seotogrow.in/sitemap.xml its made with the same plugins you are using. There should never be a different sitemap for post and pages. Infact, you should have all in one.
    and robots.txt should be something like this
    after you have rebuilt your sitemap, refresh it in your webmaster tools. let me know if i could be of any assistance. i would be glad to be of help.

    Ermmm… i’m still confused..
    my sitemap is: http://dateme.bg/sitemap_index.xml Is this ok?
    I deleted and added again my website to the webmaster tools, tried again, but the problem still appears.. i receive the same message :/

    now, after i deleted the robots.txt file from my root, in the Editor of SEO Yoast it says:


    If you had a robots.txt file and it was editable, you could edit it from here.

    first thing, the sitemap url is wrong.
    it should be http://dateme.bg/sitemap.xml after you delete or rename the robots.txt it will auto generate a new one but you will still have to make a robot.txt using notepad and upload it to the root of the directory.

    i made that. my robots file now looks like:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/

    Sitemap: http://www.seotogrow.in/sitemap.xml
    Sitemap: http://www.seotogrow.in/sitemap.xml.gz

    created it with notepad and uploaded it ot my root directory.
    When i go to the XML Sitemaps and click on the XML tab to see it, it shows the same old thing.. but it’s not like your sitemap, it’s just a plain text over white background that has no space between the sentenses..

    http://dateme.bg/post-sitemap.xml 2013-03-19T09:10:48+00:00 http://dateme.bg/page-sitemap.xml 2013-03-19T11:24:43+00:00 http://dateme.bg/category-sitemap.xml 2013-03-19T09:10:48+00:00 http://dateme.bg/post_tag-sitemap.xml 2013-03-19T09:10:48+00:00

    this could be solved in less than 5 minutes if you are willing to come on teamviewer. it seems like a problem of settings within your plugin…the sitemap url is all wrong. it should be one sitemap which should have all the pages, posts, categories, tags, and static pages.

    or else, uninstall the plugin and delete the files associated with it. re-install the plugin and then keep the default settings, you would get what you want. Dont make separate sitemaps for post and pages.

    may i have your ID on teamviewer? or somebody that wants to help? 🙂

    499 053 651 id
    5628 pwd

    I will use my panel and show you the settings..



    Forum Moderator

    Please note that these are community forums and all support should take place on the forums so that others can benefit.

Viewing 15 replies - 1 through 15 (of 24 total)
  • The topic ‘Robots.txt file problem using WordPress SEO by Yoast’ is closed to new replies.