Thank you Samboll you replies are helpful and I learned alot reading the link you posted.
I wonder how does a bot crawl a dynamic site like WP? Since WP pages are not static and on a server directory somewhere how does the bot see the content sitting in a database? Does anyone know?
Mike
nothing sees the database. Remember that all viewers, including bots, see the assembled pages. Nobody gets a direct view of the db 🙂
Ok now I am thorougly confused if there are no assembeled pages then what does a bot look at to index a site?
Mike
Depends. If it’s a well-behaved bot, it’ll crawl over the homepage, and follow links until it runs into the end of the world or NOINDEX instructions, and/or consult the robots.txt file, and/or consult the sitemap file (where available).
Just remember that “dynamic” is not the same as “constantly changing” (i.e. of place or name). So, even if a page is generated on request, its address has been established previously, which makes it a predictable entity. Which, in the end, is what bots need so as to do their job.
The pages are assembled whenever they are requested. Doesn’t matter if it is a bot or someone with a browser.
Another fantastic link around WordPress SEO:
http://yoast.com/articles/wordpress-seo/
