Jetpack has come a long way, and I include it in my Must Have Plugins list because it lowers the bar for so many features that bloggers need and would require many plugins and hours of work to set up with Jetpack, including:
1 - Contact forms that don't require captcha
2 - Social Sharing icons on posts and/or pages
3 - Automatic publishing of posts to your social networking sites
4 - Subscribe to RSS by email
5 - Publish to your blog by email
6 - Very very cool WordPress galler display options and lightbox
7 - Custom CSS so you can style your theme without messing with the themes files
8 - Lots of extra widgets
Things I would use with reservation:
1 - Photon - it caches your images to the cloud thus speeding up your site. However, depending on what other image plugins you are using and how you have inserted your images, you can have weird things happen to your images. Simply de-activate Photon if there's a problem.
2 - WP Stats - I prefer Google Analytics, but I would use this for simpler sites.
In terms of the complaints by other users, I would respond that you only have to use the parts of Jetpack you want to use. Not all features are turned on automatically.
For more complex sites, I don't use Jetpack, because I need more control over styling of the various features above and don't want redundant plugins. But for basic sites, I think it's quite wonderful!