WordPress.org

Ready to get started?Download WordPress

Forums

robots.txt (9 posts)

  1. graham85
    Member
    Posted 1 year ago #

    hi there hoping for a bit of help. just connecting a new site google and went to fetch googlebot and it comes denied by robots. so i ran the blocked urls check and it came back fine it said googlebot is allowed. any reasons why this mite be. im using yoast seo plugin aswell that didnt help.
    thanks in advance

  2. Krishna
    Volunteer Moderator
    Posted 1 year ago #

    just connecting a new site google and went to fetch googlebot and it comes denied by robots. so i ran the blocked urls check and it came back fine it said googlebot is allowed.

    Looks contradictory and confusing. If fetch as Google failed, it does not mean robots.txt blocked it.

    How old is your site? How many URLs are there in your sitemap?

  3. graham85
    Member
    Posted 1 year ago #

    just trying to get it index as it new, theirs 14 pages so far and it wont accept my xml sitemap generated by yoast it says url restricted by robots.txt.
    and yes its very confusing never had this problem before. especially when google blocked url check says it allows googlebot and then the fetch googlebot says its been denied.

  4. Krishna
    Volunteer Moderator
    Posted 1 year ago #

    Did you allow search engines index your site or disallow in your privacy settings (Dashboard)?

  5. graham85
    Member
    Posted 1 year ago #

    yeh i did at first whilst was building pages and stuff then switched it back to allow search engines

  6. Krishna
    Volunteer Moderator
    Posted 1 year ago #

    Look at your site's root and delete the robots.txt file, if you can find it and resubmit your sitemap.

  7. graham85
    Member
    Posted 1 year ago #

    i cant find the robots.txt file any where in my cpanel manager. the only thing i can think off is its been bad install as a addon domain.

  8. Krishna
    Volunteer Moderator
    Posted 1 year ago #

    If it is an addon domain site, you may look at its directory (sub directory) and add a robot.txt file as you find in this link:
    http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress

  9. graham85
    Member
    Posted 1 year ago #

    ive tried that but no look so far.
    the link http://bopinternet.co.uk/robots.txt
    ive even tried putting this in the header.php
    <?php if(is_single() || is_page() || is_category() || is_home()) { ?>
    <meta name="robots" content="all,noodp" />
    <?php } ?>
    might have to reinstall the addon domain because something not right and ive never had this problem before with an addon domain

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags