WordPress.org

Ready to get started?Download WordPress

Forums

robots.txt file for WordPress (3 posts)

  1. mAsT3RpEE
    Member
    Posted 1 year ago #

    Please help me finish this tutorial. If there is anything I've left out tell me:

    Original

    So far this is what I have:

    robots.txt
    --------------

    User-agent: *
    Disallow: /feed
    Disallow: /goto
    Sitemap: http://www.mast3rpee.tk/sitemap.xml
    Host: mast3rpee.tk
    Crawl-delay: 4

    robots.php
    --------------

    <?php
    $start = "2013/04/01";	// Date you started your blog YYYY/MM/DD
    $average = 30;			// Number of posts you make per month
    $sitemap = "http://mast3rpee.tk/sitemap.xml";
    $host = "mast3rpee.tk";
    // Prevent any type of Y2K bug. $old always = false for last year of Epoch.
    // I've stopped adding this code in my scripts because of 64 bit which will
    // defenetely stop all forms of Y2K Bugs   8P
    // Is blog old enough
    $old = ($average/30)*(time()-strtotime($start)) > 3600*24*360? true : false;
    // Output proper headers
    header ('Content-Type: text/plain');
    header ('Cache-Control: private, pre-check=0, post-check=0, max-age=36000');
    header ('Expires: ' . gmstrftime('%a, %d %b %Y %H:%M:%S GMT', time() + 36000));
    header ('Last-Modified: ' . gmstrftime('%a, %d %b %Y %H:%M:%S GMT', time() - 36000));
    if ($old) { $custom = date("Y/m");
    echo <<< ROBOTS
    User-agent: *
    Disallow: /feed
    Disallow: /goto
    Disallow: /$custom
    Sitemap: $sitemap
    Host: $host
    Crawl-delay: 4
    ROBOTS;
    } else { $custom = date("Y");
    echo <<< ROBOTS
    User-agent: *
    Disallow: /feed
    Disallow: /goto
    Disallow: /archives
    Disallow: /tag
    Disallow: /$custom
    Sitemap: $sitemap
    Host: $host
    Crawl-delay: 4
    ROBOTS;
    }
    exit; ?>

    .htaccess
    ------------

    # BEGIN Robots
    <IfModule mod_rewrite.c>
    <FilesMatch "^robots.(txt|php)$">
    Header Set Last-Modified "Fri, 01 Jan 2010 12:00:00 GMT"
    </FilesMatch>
    RewriteEngine On
    RewriteBase /
    RewriteRule ^(robots)\.txt$ /$1.php [L]
    </IfModule>
    # END Robots
  2. mAsT3RpEE
    Member
    Posted 1 year ago #

    Thank you mysterious man who fixed my post! It was broken and I couldn't edit it. Whoever you are mystery admin. Your the best!

    [You're welcome - although quite a few of us are actually female. :-)]

  3. mAsT3RpEE
    Member
    Posted 1 year ago #

    UpDate: The current format is now located at: http://mast3rpee.tk/mast3rpees-tutorials/robots-txt-file/
    And the permalink is: http://mast3rpee.tk/?p=127

Topic Closed

This topic has been closed to new replies.

About this Topic