How to prevent access download section

How do I prevent access to my upload structure

http://www.gwcc.org.nz/wp-content/uploads/2013/11/somefile.pdf

with this link, user can than navigate my entire Upload structure and possibly go somewhere I don’t want them to reach.