1) the actual document of a blog post is created as the visitor comes to the site. The click on the url causes wordpress to take the post content from the database, put it into the theme template and apply the stylesheet.
….. so the robots instructions are not in the post.
If you look in the theme files you will see the php code to create the meta tags in the page. They are probably in a file called header.php
2) I assume you mean directories when you say catalogue. The robots.txt file should be in the root directory of your blog – that is to say the directory called when the url http://www.yourdomain.com is clicked
Thread Starter
pelia
(@pelia)
1) OK, I tried adding the noindex stuff in the name tag. From start it looked like this:
<meta name=”generator” content=”WordPress <?php bloginfo(‘version’); ?>”
It ended up looking like this:
<meta name=”robots” content=”noindex,nofollow”>
<meta name=”generator” content=”WordPress <?php bloginfo(‘version’); ?>” />
Do you think this is correct? Should I just add a new line like this?
2) Yes, I meen directoris. OK, that would be in the httpdocs, because then I can see the txt file if i enter my domainname.domain/robots.txt in my browser. But wich directories should I type? Now I have this:
User-agent: *
Disallow: /wp-content/
Disallow: /index.php
Disallow: /wp-content/themes/
Disallow: /wp-content/themes/index.php
Disallow: /wp-content/themes/durable/
Disallow: /wp-content/themes/durable/index.php
Disallow: /wp-admin/
Disallow: /wp-includes/
So, I just wrote the first level of directories and also, I tried writing som single files in a manner I made up, with the slashes. Are those unnecessary or perhaps incorrect? Should I also add subdirectories?
Many thanks for your help!
Yes, adding new lines like that is fine, but your new meta tags should end with /> to be proper XHTML. Look at the existing lines and do like they do.
And if you don’t want any of your site indexed, all you need is this:
User-agent: *
Disallow: /
When you disallow one thing, everything under it is disallowed as well.
Thread Starter
pelia
(@pelia)
Many thanks for your helpful advice!