Restricting Access

I wanted to restrict media access to all but my website visitors…So I created a Bucket Policy by restricting access to a certain HTTP referrer and made sure that only the URLs which are referred from my website can get the object from s3 and show in browser (documented here). But after installing and activating this plugin, an “access denied” was shown on my plugin settings page and files weren’t being stored on s3 account. So I removed the policy and now anyone having the URL can see the object. How to restrict access?