WordPress.org

Ready to get started?Download WordPress

Forums

Robots.txt for WordPress, what to allow and disallow? (3 posts)

  1. alphakilo615
    Member
    Posted 7 months ago #

    Hello all,

    I have some questions about building a robots.txt for my wordpress site. I am not sure what I need to allow and disallow. I Googled and found some articles and used what some suggested, but I believe my site still isn't being crawled successfully. Here is a link to my robots.txt if you want to see, http://www.attorneyvidbox.com/robots.txt

    Is there a general robots.txt that wordpress provides? I just want Google to crawl my posts on my site.

    Regards,
    Alex

  2. Samuel Wood (Otto)
    Tech Ninja
    Posted 7 months ago #

    You don't actually "need" to disallow anything or do anything special for a robots.txt on a WordPress site. It can help a bit if you don't want, say, your images indexed by Google or something, but having no robots.txt is perfectly acceptable too.

    Those "Allow:" lines are entirely pointless.

  3. alphakilo615
    Member
    Posted 7 months ago #

    That's what I thought also. I use Google Webmasters and I see a lot of errors, because it crawls Authors and etc all the time. Isn't that not hurting Google web crawlers/

Reply

You must log in to post.

About this Topic