Restricted attachments

I want to put some PDF attachments on a restricted page (minutes of meetings for members only). It’s fine for the links, but can I also restrict the upload?

I tried giving the directory containing the files a protection 700 and got a nice 404 message from WordPress. But if I don’t do that, then it’s not properly secure. Anyone who finds out the address can download the PDF.