The short of it is that all the main authentication functions are in pluggable.php, which means that all those functions are “pluggable”; they can be replaced by your own functions in a plugin or whatever.
So you essentially have to write replacements for all those functions, only your functions will use your authentication system.
If you don’t know PHP too well, then you probably can’t do it without learning a lot more PHP. And if you do know PHP pretty well (along with your own auth system), then just looking at those few functions is all the documentation you’ll really need.
It’s not a minor project, mind you. To my knowledge, nobody’s ever really done it. But it’s at least possible, in theory.
Actually, as I understand it WP United combines a lot of the phpBB and WP functionality, including user tables and authentication. This may not be what you want (it sounded like you wanted to completely replace/remove the WP user table), but it should be a good place to start, or a reference if you get stuck.
I’m looking to do this as well, but I want to replace the WP authentication with that of the Invision Power Board.
The makers of IPB have actually made this easier by creating a new platform called Converge, which can centralize the login authentication for many different applications using XML-RPC method calls and method responses.
It’s just a matter of replacing the WP authentication functions with these XML-RPC calls. Unfortunately I don’t have the necessary PHP knowledge to make this happen.
If someone wants to take a shot at it, here’s the Converge documentation: http://forums.invisionpower.com/index.php?act=attach&type=post&id=11100, and I can provide the “coverge_local” files for examination.
