Starting with version 2.0 of the plugin (released on Monday night) you can upload any format FFMPEG can read (including WMVs) and you can then choose to embed one of the transcoded files instead of the original video, but that's only if the Encode button works.
I'd be surprised if the shortcode problem was unique to my plugin. Try changing your theme to something like Twenty Eleven and see if you still have that problem. If you do, then your theme is not pulling the content correctly. It's probably using get_the_content in the loop instead of the_content
The encoding issue is almost definitely related to my plugin. When it works it's not running using nice, but I'm limiting the threads and in my experience it does run nicely, without tying up the server. I'll try using nice though, it's a good idea since I'm pretty much giving up on getting anything to work well in Windows.
I can't tell yet what's causing your encoding problems. The first step is to turn on debugging in WordPress. Add the following lines to your wp-config.php file, then try to encode something and check the debug.log file in the wp-content folder.
* This will log all error notices and warnings to a file called debug.log in
* wp-content (if Apache does not have write permission, you may need to create
* the file first and set the appropriate permissions (i.e. use 666) )