Restrict appearance menu to network admin only

To remove the appearance menu from the sidebar I used remove_menu_page(). Easy!

In the documentation of this function it states:

Please be aware that this would not prevent a user from accessing these screens directly. Removing a menu does not replace the need to filter a user’s permissions as appropriate.

So I also need to filter the current user’s permissions somehow. I can not figure how to do this or what hook to use.