Robots.txt for WordPress, what to allow and disallow? (3 posts)

  1. alphakilo615
    Posted 2 years ago #

    Hello all,

    I have some questions about building a robots.txt for my wordpress site. I am not sure what I need to allow and disallow. I Googled and found some articles and used what some suggested, but I believe my site still isn't being crawled successfully. Here is a link to my robots.txt if you want to see, http://www.attorneyvidbox.com/robots.txt

    Is there a general robots.txt that wordpress provides? I just want Google to crawl my posts on my site.


  2. You don't actually "need" to disallow anything or do anything special for a robots.txt on a WordPress site. It can help a bit if you don't want, say, your images indexed by Google or something, but having no robots.txt is perfectly acceptable too.

    Those "Allow:" lines are entirely pointless.

  3. alphakilo615
    Posted 2 years ago #

    That's what I thought also. I use Google Webmasters and I see a lot of errors, because it crawls Authors and etc all the time. Isn't that not hurting Google web crawlers/

Topic Closed

This topic has been closed to new replies.

About this Topic