Ok, after looking at things, it's not going to be very simple. But, I know what direction to point you in. BuddyPress does not appear to use custom post or taxonomy types (AFAIK), which means the extension plugin I spoke of before is necessary.
I have a skeleton extension plugin together, that should get you going. It should add the group (non-linked) to the breadcrumb trail for the group home page. At the moment I do not have a BuddyPress testbed to test this. A ton of the BuddyPress code is not covered in the BuddyPress codex, so it takes some searching to figure out what each function does.
You can find what I have so far at: http://plugins.svn.wordpress.org/breadcrumb-navxt/branches/BP/breadcrumb_navxt_bp.php
Feel free to modify this to suit your needs. And, if you get something that works, and want to share it, I'll include any suggestions. Once I get a BuddyPress testbed I'll be able to fully dive into this.
Right now the "hit list" is "full" WP3.0 support (finally make "global" breadcrumb trails possible for multi-site setups plus custom post support) then BuddyPress support.