So I was mistaken. It turns out that I am seeing the same exact problem. I'll spare the details as to why I didn't recognize the behavior the first time around but here's what I'm seeing now:
I'm using a post form with custom post fields to insert data into several wp_postmeta fields. I have a separate form that I use to edit the post, including the custom post fields. When I update the post, with the edit form, the custom post fields create new meta-key/meta-value fields in wp_postmeta instead of editing the existing meta-key/meta-value data. That's actually kind of what I would expect given the nature of the wp_postmeta table.
So I changed my post form as follows. On each of the custom post fields I checked the "Unique Custom Field" checkbox. I did not make any changes to the edit post form. When I add a post with the post form everything works fine. If I edit the post the original meta-values in the wp_postmeta table are modified instead of having new meta-key/meta-value pairs added. It should be problem solved but unfortunately, editing the post adds a bunch of unexpected data to the meta-value which totally breaks everything (my browser can't even render the post afterwards). Here's an example of the meta-key and meta-value before and after an edit:
If I modify the edit post form and set all of the custom post fields to "Unique Custom Field", I still see the same effect. I have yet to determine why this happens but I'm still testing various configurations.