Support » Everything else WordPress » Robots.txt for WordPress, what to allow and disallow?

  • alphakilo615


    Hello all,

    I have some questions about building a robots.txt for my wordpress site. I am not sure what I need to allow and disallow. I Googled and found some articles and used what some suggested, but I believe my site still isn’t being crawled successfully. Here is a link to my robots.txt if you want to see,

    Is there a general robots.txt that wordpress provides? I just want Google to crawl my posts on my site.


Viewing 2 replies - 1 through 2 (of 2 total)
  • Moderator Samuel Wood (Otto)

    (@otto42) Admin

    You don’t actually “need” to disallow anything or do anything special for a robots.txt on a WordPress site. It can help a bit if you don’t want, say, your images indexed by Google or something, but having no robots.txt is perfectly acceptable too.

    Those “Allow:” lines are entirely pointless.



    That’s what I thought also. I use Google Webmasters and I see a lot of errors, because it crawls Authors and etc all the time. Isn’t that not hurting Google web crawlers/

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Robots.txt for WordPress, what to allow and disallow?’ is closed to new replies.