Inactive Themes and Header Images

Do you need to activate a theme before you can choose a different image for the header?

The problem is, I can select a different image for the header as long as the theme is inactive, the images in media just won’t load.