WordPress.org

Ready to get started?Download WordPress

Forums

[Plugin: Google XML Sitemaps] Does robots.txt need any more in it than the pointer to the sitemap? (1 post)

  1. MoonbatWingnut
    Member
    Posted 6 years ago #

    I installed Google XML Sitemaps and it added a robots.txt that looks like this:

    # BEGIN XML-SITEMAP-PLUGIN
    Sitemap: http://wendelbrume.com/sitemap.xml
    # END XML-SITEMAP-PLUGIN

    So it sends the robots to the sitemap, which I've set to contain only the pages I want crawled. I see a lot of discussion around the web about how important it is to add directions to the robots.txt file to give direction to the spiders. But since robots.txt directs them to the sitemap, and it has only the pages I want crawled, am I done?

    Thanks,

    TW

Topic Closed

This topic has been closed to new replies.

About this Topic