Robots.txt | WordPress.org

DanielThompsonUK
(@danielthompsonuk)

12 years, 12 months ago

I am having some issues regarding robots.txt when I checked my site over using seoquake I noticed it didn’t have a robots.txt. I’m wondering why this could be as I read wordpress automatically creates one. I have made sure I am set to indexable by search engines in the privacy settings but still I get a 404 page error when I try and access mydomain.com/robots.txt

Does anybody have any clues for how to fix this?

Many Thanks

Viewing 4 replies - 1 through 4 (of 4 total)

Krishna
(@1nexus)

12 years, 12 months ago

That happens when you do not have a robots.txt file. Have you ever created and uploaded one?

Thread Starter DanielThompsonUK
(@danielthompsonuk)

12 years, 12 months ago

Hey thanks for your response and nope I thought wordpress did it automatically? Is there any tutorials on here for how to upload one?

Thanks
Krishna
(@1nexus)

12 years, 12 months ago
```
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
```
Just copy and paste the above code in a plain text editor like Notepad and save it as robots.txt and upload it to your root directory.
Krishna
(@1nexus)

12 years, 12 months ago

Here are some links that will give you more info:
http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress#Robots.txt_Optimization
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag
http://www.robotstxt.org/faq.html

Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘Robots.txt’ is closed to new replies.

Tags

In: Fixing WordPress
4 replies
2 participants
Last reply from: Krishna
Last activity: 12 years, 12 months ago
Status: not resolved

Topics

Topics with no replies

Non-support topics

Resolved topics

Unresolved topics

All topics