I have one Page (as in a WP Page) on my site that I don't want crawled. If the page is http://www.mysite.com/test/ and I set my robots.txt file so the the /test/ directory isn't crawled, will this work since it's a virtual directory?
Question about robots.txt (4 posts)
-
-
The crawling is done by links so it should be fine if I understand your question correctly. You can also use is_page() in the header to set a "no index" meta tag for particular pages.
-
Yes, it will work.
::Darn, too slow again.::
-
Cool, thanks guys.
Topic Closed
This topic has been closed to new replies.
