Comment visability on a user basis

I would like to stop certain users from being able to see and add comments on my site.

The way I currently have the blog setup, I upload new graphic design work as a new post. Each client needs to login and can only view posts within their own category, where they can use the comments to request changes and approve work.

Now I have setup a user login for people wishing to see some of my work as a sort of portfolio. However, I don’t want them to see comments made by clients, nor do I want them to be able to add comments to the work.

Does anyone know how I can set this up? I’ve tried quite a few different Role/Capability manager plugins, but I can’t seem to find one that has comment viewing as a changeable capability.