edit my CSS so I can add some small buttons below the header but above the top navigation bar.
You might need to edit the header.php to add the navigation items presented via buttons and add styling for new items in your theme's style.css
the forum system I am going to use is BuddyPress.
BuddyPress is a plugin for social networking. The forum which can be installed internally via BuddyPress is bbPress forum.
Also, if you have any tips for the sub-blog, please don't hesistate to say.
It depends on when you want to implement this project. If you could wait till May or so of this year for WordPress 3.0, you could easily set up a sub-blog and activate BuddyPress in that sub-blog instead of in the main site. In addition, you would only need one installation of WordPress, WP 3.0 that is, to manage all the sub-blogs you would like to add.
If you want to implement the project now, you could install BuddyPress if you have WP 2.9.2 and then add the navigation to the forums, groups, members, etc. in that new navigation block with buttons you want to add. Be aware that you might have to use the BuddyPress Template Pack and will have to tweak files and styles to match with your current theme's layout.