It's a little confusing if you're brand new to WP tags, but not too hard once you get the hang of it.
Like moshu says, open your sidebar.php in a plain text editor, like Notepad or Wordpad. (not MS Word, or it will mess with your code!!!)
Look for this bit of code:
<?php wp_list_cats('
That the beginning piece, it will be followed (maybe) by various parameters. If so, add the following just after the single quote above:
hide_empty=0&
But if it has no existing parameters listed -- meaning that there is nothing between the single quotes/parenthesis -- then you'll add everything I listed except the ampersand (&) mark, and would look like this:
<?php wp_list_cats('hide_empty=0'); ?>