Control what post types meta is added to

I’m running into some memory overflow errors on the /media endpoint, and I’m realizing I don’t actually need the yoast output on that endpoint at all. Is there some way to tell this plugin not to include it on specific post types, either by filter or configuration?