Support » Fixing WordPress » robots.txt

  • hi there hoping for a bit of help. just connecting a new site google and went to fetch googlebot and it comes denied by robots. so i ran the blocked urls check and it came back fine it said googlebot is allowed. any reasons why this mite be. im using yoast seo plugin aswell that didnt help.
    thanks in advance

Viewing 8 replies - 1 through 8 (of 8 total)
  • Krishna

    (@1nexus)

    Member

    just connecting a new site google and went to fetch googlebot and it comes denied by robots. so i ran the blocked urls check and it came back fine it said googlebot is allowed.

    Looks contradictory and confusing. If fetch as Google failed, it does not mean robots.txt blocked it.

    How old is your site? How many URLs are there in your sitemap?

    just trying to get it index as it new, theirs 14 pages so far and it wont accept my xml sitemap generated by yoast it says url restricted by robots.txt.
    and yes its very confusing never had this problem before. especially when google blocked url check says it allows googlebot and then the fetch googlebot says its been denied.

    Krishna

    (@1nexus)

    Member

    Did you allow search engines index your site or disallow in your privacy settings (Dashboard)?

    yeh i did at first whilst was building pages and stuff then switched it back to allow search engines

    Krishna

    (@1nexus)

    Member

    Look at your site’s root and delete the robots.txt file, if you can find it and resubmit your sitemap.

    i cant find the robots.txt file any where in my cpanel manager. the only thing i can think off is its been bad install as a addon domain.

    Krishna

    (@1nexus)

    Member

    If it is an addon domain site, you may look at its directory (sub directory) and add a robot.txt file as you find in this link:
    http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress

    ive tried that but no look so far.
    the link http://bopinternet.co.uk/robots.txt
    ive even tried putting this in the header.php
    <?php if(is_single() || is_page() || is_category() || is_home()) { ?>
    <meta name=”robots” content=”all,noodp” />
    <?php } ?>
    might have to reinstall the addon domain because something not right and ive never had this problem before with an addon domain

Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘robots.txt’ is closed to new replies.