WordPress Leaves Empty Blocks When Removing Widgets

Whenever I remove the last widget from a sidebar in Widgets, it leaves an empty block with a plus symbol whenever the block I removed was at the bottom. I cannot find a way to delete these rectangles with pluses in the middle. The end result are empty blocks rendering on the sidebar.

How do I delete these empty blocks? Right now the only option is to click them and choose a block to add.