It is not a bug, that is what the feature "Show only specific number of questions" is supposed to do. It was made that way by design.
This feature makes taking the test fair, in that everyone who takes the quiz will be asked the same questions, from the larger question pool.
Also note that some questions can have more point value then other questions, so using just a random question feature might not be fair for everyone who takes the quiz. Some people could get the easy questions and others could get the harder more point value questions.
If you want WP PRO Quiz to always generate a random question quiz, from the same question pool, then you might want to make a donation and ask for that feature. I agree, it would be a nice feature to have, if the random questions were distributed fairly.
Xeno010 does read the comments on this forum and implements some of the suggestions that people ask for. If you browse the message board, you will see a lot of requests for features.