There are supposedly security improvements. Plus you'll probably want to upgrade in the long run anyway. The general advice is always upgrade.
You can read the changelog or even just planet.wordpress.org for recommendations from true WP experts and insiders (not me) who will invariably tell you to upgrade asap.
That said, I'm sure there are 100's if not millions of blogs running WP 2.5, 2.3, and even 1.5.3 still, and they mostly seem to survive. Depends how risky you want to be & if you keep backups.
Why is that button so important? Just wondering, I don't even use that button myself, nor the media uploader. There's lots of ways to get pictures in your blog.