Limiting access to certain categories

Is it possible to limit access on what categories a certain user can view? For instance, if a user is part of category “A” but not category “B”, the user would only be able to view “A” and not “B”.

I know WordPress has built in permissions but it looks like it’s limited to excluding access by roles or by member. The latter mentioned would work in what I’m attempting to accomplish but every user would need to be highlighted for a particular category and the field box to enable this feature does not display the category assigned to a user with the connections business directory plugin, making the process an extended one.