Support » Plugins and Hacks » robots.txt, sitemap.xml question

  • I was wondering if anybody knows how to allow bots and spiders to see private posts or content. Is there something I can add in the sitemap.xml or robots.txt files. Also would this work for pay only content using easypaypal or arepaypal plugins?

    Thanks for the help

Viewing 4 replies - 1 through 4 (of 4 total)
  • Bots/spiders see exactly what a person can see. If the content is password protected by WordPress, a bot can’t see it.

    well.. hallsofmontezuma is right about the bots.

    But the sitemap.xml is different, you can add whatever you want there , (even manually) and the tools (like google’s) will be able to see THE LINK ITSELF, even if not what’s on it , but they will know it exist.

    Search engines don’t like it when you have a link to something they can’t see.

    hallsofmontezuma and krembo99 – Thanks for the info, so how do large, commercial websites with pay-only content still manage to have high google rankings, if the crawlers can’t see it?

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘robots.txt, sitemap.xml question’ is closed to new replies.