At the time there’s no option for the toolbar customization; if you want to display only a mute/unmute button you can hide the toolbar unchecking the “display controls” check and create your own control somewhere in your page as follow:
<button class="command" onclick="jQuery('#bgndVideo_home').setYTPVolume(100)"> switch volume </button>
Where the ID is:
#bgndVideo_home -> in the case the video is defined as home video.
#bgndVideo1 -> if you set it via the tinyMCE shortcode window.
This will create a button that switch the volume from 0 to the maxVol.