WordPress.org

1. peps2004
   Member
   Posted 4 months ago #

   I'm wanting to stop search engines from crawling a page I'm going to be setting up in the next few days and as far as I'm aware this means I need to upload a robots.txt file.

   The thing is, I'm not actually sure how this is done. Is it simply a case of pasting the following code into the CSS style sheet:

   User-agent: *
   Disallow: /theurlthatidontwantcrawled

2. blogjunkie
   Member
   Posted 4 months ago #

   No, you don't paste it into your CSS file. Create a new text file called robots.txt in the root of your site, i.e. example.com/robots.txt and paste the code from above in that file. You'll need to do this with a FTP client.

   Here's an example file for your reference: http://www.google.com/robots.txt

3. peps2004
   Member
   Posted 4 months ago #

   Hi Blogjunkie,

   Thanks for the reply. How would I go about doing this with an FTP client? Sorry if this seems like a dumb question, but we update everything online rather than in a separate program.

4. blogjunkie
   Member
   Posted 4 months ago #

   Hmm sorry it's too difficult to explain how to use FTP client here. Try searching YouTube for a FileZilla tutorial. FileZilla is free FTP client for windows and Mac. Good luck

Reply

You must log in to post.