I'm not 100% certain on it since I haven't tested it fully yet, but...
.htaccess files are additive. Whenever you request a page, the webserver basically goes through every directory down the tree from the root (specified by the closest match of <Directory ...> in the httpd.conf file), and adds all the .htaccess files together. As it traverses them, it parses each one. Later .htaccess files override previous ones, but only for the same specified items. RewriteRules are cumulative.
So what I think is going on is that the authorization in the password protected directory is forcing a 401 response ("Authorization Required") back to the client. Normally, the client would get the 401 and ask for a password.
However, in this case, this 401 response is intercepted by the WordPress RewriteRules which says to rewrite everything to WordPress. This is because .htaccess's are cumulative and your closest matching Directory is the root.
So, by forcing an ErrorDocument for the 401 response before the WordPress rules, you pre-empt them (since the file actually exists, the RewriteRules won't take effect upon it), causing your 401 document to be sent instead of rewriting to WordPress. However, the client doesn't give a crap about that document, it sees the 401 and asks for a password.
Notice that if you fail to give a password three times (depending on the client), you'll probably get forwarded back to WordPress. This may or may not be what you want. If you add the 403 line, you'll get either your error document or a 403 Forbidden page back, not certain which.
Other possible solutions:
- Add a new Directory statement to httpd.conf, specifically specifying your password protected directory, thus bypassing the wordpress rewrites from the htaccess search path.
- Add another rewrite to the top of the wordpress rewrites that pre-empts them for that directory only.
But forcing a 401/403 document seems like the best solution to me, since it will work with any password protected subdirectories you care to add, without having to specify them.
Did that make any sense? ;)