WordPress.org

Ready to get started?Download WordPress

Forums

Disallowing Permalink'd page in robots.txt (4 posts)

  1. robertallen
    Member
    Posted 1 year ago #

    I have custom permalinks plugin and I have set a page like so...

    http://www.mydomain.com/thisisapage1/

    I was wondering, is it just as simple as adding
    Disallow: /thisisapage1/

    in the robots.txt if I don't want it crawled?

  2. tomontoast
    Member
    Posted 1 year ago #

    That is correct but this will also disallow robots from accessing pages in the same structure as '/thisisapage1/' eg '/thisisapage1/foo/' will also be disallowed

  3. robertallen
    Member
    Posted 1 year ago #

    Thank you very much!

    Should I also disallow /tag/ and /category/ ? I'm guessing that helps to avoid duplicate content as the blog posts already have their own URL's, right?

  4. tomontoast
    Member
    Posted 1 year ago #

    I'm no SEO expert, in my opinion search engines recognise tag and category pages and won't penalise you for using them so I wouldn't worry.

Topic Closed

This topic has been closed to new replies.

About this Topic