Support » Fixing WordPress » robots.txt file for WordPress

  • mAsT3RpEE

    (@mast3rpee)


    Please help me finish this tutorial. If there is anything I’ve left out tell me:

    Original

    So far this is what I have:

    robots.txt
    ————–

    User-agent: *
    Disallow: /feed
    Disallow: /goto
    Sitemap: http://www.mast3rpee.tk/sitemap.xml
    Host: mast3rpee.tk
    Crawl-delay: 4

    robots.php
    ————–

    <?php
    $start = "2013/04/01";	// Date you started your blog YYYY/MM/DD
    $average = 30;			// Number of posts you make per month
    $sitemap = "http://mast3rpee.tk/sitemap.xml";
    $host = "mast3rpee.tk";
    // Prevent any type of Y2K bug. $old always = false for last year of Epoch.
    // I've stopped adding this code in my scripts because of 64 bit which will
    // defenetely stop all forms of Y2K Bugs   8P
    // Is blog old enough
    $old = ($average/30)*(time()-strtotime($start)) > 3600*24*360? true : false;
    // Output proper headers
    header ('Content-Type: text/plain');
    header ('Cache-Control: private, pre-check=0, post-check=0, max-age=36000');
    header ('Expires: ' . gmstrftime('%a, %d %b %Y %H:%M:%S GMT', time() + 36000));
    header ('Last-Modified: ' . gmstrftime('%a, %d %b %Y %H:%M:%S GMT', time() - 36000));
    if ($old) { $custom = date("Y/m");
    echo <<< ROBOTS
    User-agent: *
    Disallow: /feed
    Disallow: /goto
    Disallow: /$custom
    Sitemap: $sitemap
    Host: $host
    Crawl-delay: 4
    ROBOTS;
    } else { $custom = date("Y");
    echo <<< ROBOTS
    User-agent: *
    Disallow: /feed
    Disallow: /goto
    Disallow: /archives
    Disallow: /tag
    Disallow: /$custom
    Sitemap: $sitemap
    Host: $host
    Crawl-delay: 4
    ROBOTS;
    }
    exit; ?>

    .htaccess
    ————

    # BEGIN Robots
    <IfModule mod_rewrite.c>
    <FilesMatch "^robots.(txt|php)$">
    Header Set Last-Modified "Fri, 01 Jan 2010 12:00:00 GMT"
    </FilesMatch>
    RewriteEngine On
    RewriteBase /
    RewriteRule ^(robots)\.txt$ /$1.php [L]
    </IfModule>
    # END Robots

Viewing 2 replies - 1 through 2 (of 2 total)
Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘robots.txt file for WordPress’ is closed to new replies.