I see you’re using P3 to run your tests. It’s worth noting that plugins like P3 run their tests while logged in. It consequently isn’t quite representative of what your visitors experience. Jetpack does a fair amount more when users, like you, are logged in which sways the results a bit.
If you’re interested, there are also some words on this from one of P3’s developers at https://wordpress.org/support/topic/not-worth-the-performance-penalties?replies=6#post-3933061
I’d recommend running your P3 tests with a manual scan, and use a different browser to see how Jetpack would impact load for a real visitor. That should give you more realistic figures.
If you still notice some performance issues, I’d recommend deactivating each Jetpack module, one at a time, until you can find the source of the issue.
You may find that modules like Sharing or Sidebar Widgets, while adding useful features to your site, also slow things down a bit as they make calls to third-party resources like Twitter, Facebook, and other services. That’s to be expected regardless of the plugin you use, and in such cases you’ll need to decide whether the added features are worth the performance hit.
I hope this helps.
