Title: WordPress and robots.txt
Last modified: August 18, 2016

---

# WordPress and robots.txt

 *  [hawk21](https://wordpress.org/support/users/hawk21/)
 * (@hawk21)
 * [19 years ago](https://wordpress.org/support/topic/wordpress-and-robotstxt/)
 * Hi, i want to exclude some urls from being indexed.
    But wordpress does not use
   html pages, so how can i specify to spiders into the robots.txt that i don’t 
   want index some of my posts?
 * THX

Viewing 4 replies - 1 through 4 (of 4 total)

 *  [Chris_K](https://wordpress.org/support/users/handysolo/)
 * (@handysolo)
 * [19 years ago](https://wordpress.org/support/topic/wordpress-and-robotstxt/#post-559295)
 * an URL is an URL, regardless of if there’s a text file laying around.
 * Just specify the post URLs in robots.txt like you normally would.
 *  Thread Starter [hawk21](https://wordpress.org/support/users/hawk21/)
 * (@hawk21)
 * [19 years ago](https://wordpress.org/support/topic/wordpress-and-robotstxt/#post-559318)
 * So i can just put this in my robots.txt
 * >  User-Agent: *
   >  Disallow: /archives/2007/postname
 * without any specification of the file type (html, php…) and it works?
 *  Moderator [Samuel Wood (Otto)](https://wordpress.org/support/users/otto42/)
 * (@otto42)
 * WordPress.org Admin
 * [19 years ago](https://wordpress.org/support/topic/wordpress-and-robotstxt/#post-559323)
 * Yes, you can.
 * A URL is a URL. It doesn’t have a “filetype”. Those extensions you’re talking
   about don’t mean the same things on the web.
 *  Thread Starter [hawk21](https://wordpress.org/support/users/hawk21/)
 * (@hawk21)
 * [19 years ago](https://wordpress.org/support/topic/wordpress-and-robotstxt/#post-559335)
 * OK, thanks.

Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘WordPress and robots.txt’ is closed to new replies.

 * In: [Fixing WordPress](https://wordpress.org/support/forum/how-to-and-troubleshooting/)
 * 4 replies
 * 3 participants
 * Last reply from: [hawk21](https://wordpress.org/support/users/hawk21/)
 * Last activity: [19 years ago](https://wordpress.org/support/topic/wordpress-and-robotstxt/#post-559335)
 * Status: not resolved

## Topics

### Topics with no replies

### Non-support topics

### Resolved topics

### Unresolved topics

### All topics