Robots.txt file – Google not indexing

  • billiesblues

    (@billiesblues)


    10 years, 2 months ago

    Hi everyone,
    I would like to ask your help: I’m a newbie and I have a free blog with wordpress (so like sample.wordpress.com). For few days I kept it offline with the option “I would like my website to be private” to ultimate it. Now now I launched it online, but if I look for it on Google I received the following error message “A description for this result is not available because of this site’s robots.txt”
    I can see a robots.txt file typing /robots.txt at the end of my site url, do you think it’s just a matter of time and it will take few days to adjust it?
    Thanks a lot and have a nice day!

Viewing 3 replies - 1 through 3 (of 3 total)
  • Clayton James

    (@claytonjames)

    10 years, 2 months ago

    I can see a robots.txt file typing /robots.txt at the end of my site url

    What does the file say now?

    do you think it’s just a matter of time and it will take few days to adjust it?

    If you changed the settings to allow it to be indexed, then yes.

    Thread Starter billiesblues

    (@billiesblues)

    10 years, 2 months ago

    Thanks for your answer. Yes, I changed the settings few days ago.
    This is the content of my robots.txt file:

    User-agent: IRLbot
    Crawl-delay: 3600

    User-agent: *
    Disallow: /next/

    User-agent: *
    Disallow: /mshots/v1/

    # har har
    User-agent: *
    Disallow: /activate/

    User-agent: *
    Disallow: /public.api/

    # MT refugees
    User-agent: *
    Disallow: /cgi-bin/

    User-agent: *
    Disallow: /wp-login.php

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Thanks again for the support!

    Clayton James

    (@claytonjames)

    10 years, 2 months ago

    I think you should be okay. However, that looks like it might be a wordpress.com site, rather than self hosted. You may want to see if you can get a sitemap url in there somewhere as well.

    https://en.support.wordpress.com/sitemaps/

    Here are some topics from the wordpress.com support forums that might help answer any questions about their robots.txt file format.

    https://en.support.wordpress.com/?s=robots.txt

    Hope that helps. Good luck!

    [edit] ..I’m such a dope. You already mentioned it was on a wordpress.com site. 🙂

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Robots.txt file – Google not indexing’ is closed to new replies.