Support » How-To and Troubleshooting » Technorati and robots.tx

Technorati and robots.tx

  • I received this message when trying to register my site with Technorati.

    “Our access is explicitly blocked by your robots.txt file.”

    My privacy settings are set to “I would like my blog to be visible to everyone, including search engines (like Google, Bing, Technorati) and archivers”

    I am not sure what to do next.

Viewing 13 replies - 1 through 13 (of 13 total)
  • esmi


    Forum Moderator

    What’s in your robots.txt file? It should be in your domain’s root folder.

    I am not sure how get to my root folder. Can you give me an idea of where it might be/what folder in the control panel for my host (Blue Host)?

    Your root folder should be the public_html folder, unless your site is installed in a subfolder, then your root folder will be something like public_html/yoursite

    You will still want to check the robots file in your public_html even if your site is in a subfolder like public_html/yoursite. There may also be a deny in the .htaccess file, so if checking the robots.txt file doesn’t help, make sure hidden files are enabled and check your .htaccess.

    I am really challenged. 🙂 I can’t find the file in Public_html folder. This is relatively new to me. Can yo give me a sample pathway?




    Forum Moderator

    Let’s try another approach – site url?


    That looks like the do_robots file from WordPress. You should create your own file that doesn’t include the ambiguous syntax used by default.

    You can access your folders using the File Manager tool within the Files section of the cPanel. Your public_html should be in there and you can edit an existing robots.txt file from within there or create your own robots.txt file from within there.

    I uploaded a file named robots.txt to the root file with the below included. Technorati is still telling me the robots.txt file is prohibiting them from crawling the site???

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /tmp/
    Disallow: /junk/

    Did I provide the wrong information?



    Forum Moderator

    That looks fine to me.

    I looked a little closer, and it appears your blog feed is invalid. Check it out:

    <link rel="alternate" type="application/rss+xml" title="RSS 2.0" href="http://feeds.feedburner.com/NaturalProductsBiz" />

    Technorati is going to take one look at that and then ignore your blog.

    How do I fix the feed?



    Forum Moderator

    It’s not the feed. It’s the feed url. You;re pointing to Feedburner instead of your own feeds. Try replacing:

    <link rel="alternate" type="application/rss+xml" title="RSS 2.0" href="http://feeds.feedburner.com/NaturalProductsBiz" />


    <link rel="alternate" type="application/rss+xml" title="RSS 2.0" href="<?php bloginfo('rss2_url'); ?>" />
    <link rel="alternate" type="text/xml" title="RSS .92" href="<?php bloginfo('rss_url'); ?>" />
    <link rel="alternate" type="application/atom+xml" title="Atom 0.3" href="<?php bloginfo('atom_url'); ?>" />

    in header.php

Viewing 13 replies - 1 through 13 (of 13 total)
  • The topic ‘Technorati and robots.tx’ is closed to new replies.
Skip to toolbar