Hiding registered blocks from inserter

In the Block Editor, is there a straightforward way to hide a block from the inserter? I have two blocks in my plugin that are part of a block template for a custom post type, and so are automatically added to new posts. Those blocks are only relevant to that post type and only make sense as being one block : one post, so I want to keep them out of the inserter but they need to be registered.

I’m sure I can do this with css or js tricks, but it’d be nice if there was a clean way to do it.