• Hi!

    Im kinda new to wordpress so the tech stuff confuses me.

    I found out that robots.txt file have its advanatages and i wanted to create one.

    However, im not quite sure if the access I defined are correct. So can anyone help me and confirm?

    Here’s what’s inside my robots.txt. (Copied from a sample here in the forum)

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Allow: /wp-content/uploads

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*

    # Internet Archiver Wayback Machine
    User-agent: ia_archiver
    Disallow: /

    # digg mirror
    User-agent: duggmirror
    Disallow: /

    Sitemap: http://cerebralfancy.co.cc/sitemap.xml

  • The topic ‘Sitemap Question?’ is closed to new replies.