This basically solved it for me:
https://wordpress.stackexchange.com/questions/344009/how-to-use-phpmailer-in-a-function-in-wordpress
Though I ended up using some globals to pass my parameters to PHPmailer, because of the bizarr way WP includes it all… (more elegant way would be to use closures).
Hope this helps anyone with similar problems.
Thanks for the hint!
And you are right – I need to change how I include the mailer in my script.