Posted 1 year ago #
I was wondering if anybody knows how to allow bots and spiders to see private posts or content. Is there something I can add in the sitemap.xml or robots.txt files. Also would this work for pay only content using easypaypal or arepaypal plugins?
Thanks for the help
Posted 1 year ago #
Bots/spiders see exactly what a person can see. If the content is password protected by WordPress, a bot can't see it.
well.. hallsofmontezuma is right about the bots.
But the sitemap.xml is different, you can add whatever you want there , (even manually) and the tools (like google's) will be able to see THE LINK ITSELF, even if not what's on it , but they will know it exist.
Posted 1 year ago #
Search engines don't like it when you have a link to something they can't see.
Posted 1 year ago #
hallsofmontezuma and krembo99 - Thanks for the info, so how do large, commercial websites with pay-only content still manage to have high google rankings, if the crawlers can't see it?
This topic has been closed to new replies.
