The solution ended up being pretty simple. Just check messages in screen options. The warning message is unrelated.
There should be a Delete button under the Update button in the user editing page.
There doesn’t seem to be any bulk editing option though…
More weird stuff:
Now the border is gone in Chrome after clearing it’s cache, but not in Firefox or IE. What’s going on?