WordPress and Joomla can happily exist like that.
You could install as many Joomlas or WordPresses as you like within subdomains or subfolders
how to actually creat the subdomain is going to be hosting specific really
Most hosts have a 1 clisk install for WordPress - you would just establish the subdomain, and install to it per their instructions
Or you could do it manually
The impiortant thing would be to first establish your subdomain, link it to your desired folder, and install WP there. NEVER install WP and Joomla, or multiple WPs in the same folder