WordPress.org

Ready to get started?Download WordPress

Forums

robots.txt, sitemap.xml question (5 posts)

  1. cartpauj
    Member
    Posted 6 years ago #

    I was wondering if anybody knows how to allow bots and spiders to see private posts or content. Is there something I can add in the sitemap.xml or robots.txt files. Also would this work for pay only content using easypaypal or arepaypal plugins?

    Thanks for the help

  2. Michael Torbert
    WordPress Virtuoso
    Posted 6 years ago #

    Bots/spiders see exactly what a person can see. If the content is password protected by WordPress, a bot can't see it.

  3. krembo99
    Member
    Posted 6 years ago #

    well.. hallsofmontezuma is right about the bots.

    But the sitemap.xml is different, you can add whatever you want there , (even manually) and the tools (like google's) will be able to see THE LINK ITSELF, even if not what's on it , but they will know it exist.

  4. Michael Torbert
    WordPress Virtuoso
    Posted 6 years ago #

    Search engines don't like it when you have a link to something they can't see.

  5. cartpauj
    Member
    Posted 6 years ago #

    hallsofmontezuma and krembo99 - Thanks for the info, so how do large, commercial websites with pay-only content still manage to have high google rankings, if the crawlers can't see it?

Topic Closed

This topic has been closed to new replies.

About this Topic