So, after a little bit more digging, I have discovered that however they were all added, I was not deleting the entire button block. I was only deleting the inner button block.
I have tested this by deleting the actual outer button block and saving the page, going back out of the page and then re-editing the page. This has fixed it.
