WordPress.org

Ready to get started?Download WordPress

Forums

WordPress SEO by Yoast
[closed] Robots.txt file problem using WordPress SEO by Yoast (25 posts)

  1. datemebg
    Member
    Posted 1 year ago #

    Hi,
    i didn't knew i should post my problem here, so there it is:

    Hi,
    i have a problem with my robots.txt file.
    I was using Google XML Sitemaps and received messages in the webmaster tools that bots blocked my pages, i deleted the plugin, deleted the sitemaps.xml file from the root directory and switched to SEO by Yoast, i enabled the XML Sitemaps.
    I tested the new sitemap in the webmaster tools and it says:

    Warnings

    Url blocked by robots.txt.

    Sitemap contains urls which are blocked by robots.txt.

    4

    Value: http://dateme.bg/category-sitemap.xml

    -

    Mar 19, 2013
    Value: http://dateme.bg/page-sitemap.xml

    -

    Mar 19, 2013
    Value: http://dateme.bg/post-sitemap.xml

    Here is my robots.txt file:
    # User-agent: *

    # Disallow: /wp-admin/

    # Disallow: /wp-includes/

    # Disallow: /wp-trackback

    # Disallow: /wp-feed

    # Disallow: /wp-comments

    # Disallow: /wp-content/plugins

    # Disallow: /wp-content/themes

    # Disallow: /wp-login.php

    # Disallow: /wp-register.php

    # Disallow: /feed

    # Disallow: /trackback

    # Disallow: /cgi-bin

    # Disallow: /comments

    # Disallow: *?s=

    I'm really sorry, but i have no idea what to do. I don't have any nofollow links or metatags..

    I will appreciate any help!
    Thank you

    http://wordpress.org/extend/plugins/wordpress-seo/

  2. seowizardaltaf
    Member
    Posted 1 year ago #

    do you have a physical file by the name of robots.txt in your root directory?

  3. datemebg
    Member
    Posted 1 year ago #

    yes, i do!

  4. seowizardaltaf
    Member
    Posted 1 year ago #

    rename the robots.txt and the Yoast plugin will make on of its own. Or else, you could go to http://yoast.com/example-robots-txt-wordpress/ and make a new robot.txt

    User-Agent: *
    Disallow: /wp-content/plugins/

    this is the simplest of all.

  5. datemebg
    Member
    Posted 1 year ago #

    Hi again,
    i'm very sorry, but i'm a newbie. How should i name the robots.txt file? You say i should rename it? What name should i give it?

  6. seowizardaltaf
    Member
    Posted 1 year ago #

    its ok, we were all newbies when we started. just rename it to test.txt. if you really need help and wish to learn then do a teamviewer session with me. We could fix it together...

  7. datemebg
    Member
    Posted 1 year ago #

    Thank you for your understanding, i just did that.
    Should i wait before i check again at the webmaster tools? Because now it shows the same results, also when i test a sitemap, may i add the main sitemap, that Yoast makes - sitemap_index.xml or should i test them separately - for the post and the page?
    I love this plugin and highly appreciate your help! I've put a lot of effort in this website, please advise what to do further :)

  8. seowizardaltaf
    Member
    Posted 1 year ago #

    We all love Yoast :)
    you should have one sitemap which should be something like this

    http://www.seotogrow.in/sitemap.xml its made with the same plugins you are using. There should never be a different sitemap for post and pages. Infact, you should have all in one.
    and robots.txt should be something like this
    after you have rebuilt your sitemap, refresh it in your webmaster tools. let me know if i could be of any assistance. i would be glad to be of help.

  9. datemebg
    Member
    Posted 1 year ago #

    Ermmm... i'm still confused..
    my sitemap is: http://dateme.bg/sitemap_index.xml Is this ok?
    I deleted and added again my website to the webmaster tools, tried again, but the problem still appears.. i receive the same message :/

  10. datemebg
    Member
    Posted 1 year ago #

    now, after i deleted the robots.txt file from my root, in the Editor of SEO Yoast it says:

    Robots.txt

    If you had a robots.txt file and it was editable, you could edit it from here.

  11. seowizardaltaf
    Member
    Posted 1 year ago #

    first thing, the sitemap url is wrong.
    it should be http://dateme.bg/sitemap.xml after you delete or rename the robots.txt it will auto generate a new one but you will still have to make a robot.txt using notepad and upload it to the root of the directory.

  12. datemebg
    Member
    Posted 1 year ago #

    Ok,
    i made that. my robots file now looks like:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/

    Sitemap: http://www.seotogrow.in/sitemap.xml
    Sitemap: http://www.seotogrow.in/sitemap.xml.gz

    created it with notepad and uploaded it ot my root directory.
    When i go to the XML Sitemaps and click on the XML tab to see it, it shows the same old thing.. but it's not like your sitemap, it's just a plain text over white background that has no space between the sentenses..

    http://dateme.bg/post-sitemap.xml 2013-03-19T09:10:48+00:00 http://dateme.bg/page-sitemap.xml 2013-03-19T11:24:43+00:00 http://dateme.bg/category-sitemap.xml 2013-03-19T09:10:48+00:00 http://dateme.bg/post_tag-sitemap.xml 2013-03-19T09:10:48+00:00

  13. seowizardaltaf
    Member
    Posted 1 year ago #

    this could be solved in less than 5 minutes if you are willing to come on teamviewer. it seems like a problem of settings within your plugin...the sitemap url is all wrong. it should be one sitemap which should have all the pages, posts, categories, tags, and static pages.

    or else, uninstall the plugin and delete the files associated with it. re-install the plugin and then keep the default settings, you would get what you want. Dont make separate sitemaps for post and pages.

  14. datemebg
    Member
    Posted 1 year ago #

    may i have your ID on teamviewer? or somebody that wants to help? :)

  15. seowizardaltaf
    Member
    Posted 1 year ago #

    499 053 651 id
    5628 pwd

    I will use my panel and show you the settings..

  16. esmi
    Forum Moderator
    Posted 1 year ago #

    Please note that these are community forums and all support should take place on the forums so that others can benefit.

  17. datemebg
    Member
    Posted 1 year ago #

    I'm really sorry buddy, my connection broke, may we try again?

  18. seowizardaltaf
    Member
    Posted 1 year ago #

    yes we can try again....
    499 053 651
    5628

  19. filipecrocks
    Member
    Posted 12 months ago #

    Hi

    I have the same problem, could you please help me?

    How can i make one sitemap with the plugin, or I have to make it manual?

    Thanks

  20. seowizardaltaf
    Member
    Posted 12 months ago #

    Hi, you could use google XML sitemap plugin which will generate a good sitemap and for robots. Txt you will have to manually make it. Let me know if I could be of any assistance.

  21. fish911
    Member
    Posted 9 months ago #

    I'm not sure if I have a issue or not if somebody could point me in the right direction it would be greatly appreciated.

    I'm trying to get yoast to display my robots.txt file for editing reasons with no luck. I've created a sitemap and submited it to google webmasters tools with no errors but when I go to yoast, edit file, and look under robot.txt i get this message " If you had a robots.txt file and it was editable, you could edit it from here".

    Here's the kicker. When I search mysite/robots.txt I'm receiving this
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Then under root, robots.txt in my file manager I have this: User-agent:
    Googlebot
    Allow: /
    Sitemap:http://tampabayroofing.jrjbroofing.org/sitemap_index.xml

    I'm not sure why my robots.txt file under root is not the same as when searched in the browser or displayed in yoast.

  22. seowizardaltaf
    Member
    Posted 9 months ago #

    you need to manually make robots.txt file using notepad to be able to edit it. you will see sitename.com/robots.txt but that will not be editable until you make one manually.

  23. fish911
    Member
    Posted 9 months ago #

    Hey seowizardaltaf thanks for the response.

    I made a robots.txt file using gedit the linux version of notepad. Then I uploaded it to my child theme directory last night.

    This is what I placed in the robots.txt file that I made:
    User-agent: Googlebot
    Allow: /

    Sitemap:http://tampabayroofing.jrjbroofing.org/sitemap_index.xml

    But it still doest appear in yoast or when searching with the browser mysite.com/robots.txt a completely different set of code is ">displayed.

    I'm really not sure what's going on.

  24. seowizardaltaf
    Member
    Posted 9 months ago #

    could you do a tv session?

  25. WPyogi
    Volunteer Moderator
    Posted 9 months ago #

    Guys, this thread is 7 months old - please start your own threads and do not take help off these forums:

    http://codex.wordpress.org/Forum_Welcome#Helping_Out

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic