If you want to exclude all the search engine spiders from your entire domain, you would add a robots.txt file your website folder and put the following text in it:
User-agent: *
Disallow: /
Use with care as the site won’t get indexed by search engines while the file is still there. See here for more information on robots.txt.
Here’s the thing, if I have installed WordPress on my hosting server that contains my current company site, will that not stop all search engine spiders and impact my current working site too? So that would probably be a bad thing overall, right?
It should only apply to subfolders, so if wordpress is installed to http://www.yourdomain.com/wordpress/ and you put robots.txt in http://www.yourdomain.com/wordpress/robots.txt, it should be fine.
To identify each file you want to exclude, and not folders and domains, you can also put this information in meta fields in the html or wordpress theme template directly, for example:
<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
More info on the meta no robots approach here.
Another idea would be to forward all traffic coming to your under-development site to the new site, using something like this:
<META
HTTP-EQUIV="Refresh"
CONTENT="5; URL=autoforward_target.html">
