you "sound" like you want to hear that it doesn't.
I'll restate my own experience.
Except for VERY minor issues, my own site does NOT display differently in different browsers. Unless of course, you include browsers that dont support a certain facet of some CSS that I have used.
There is one safari issue that I am aware of that has nothing to do with validation.
Firefox is forgiving - far more forgiving than IE. Thats precisely WHY there are hundreds of posts here where someone's site looks great in it, but not in *some* version of IE.
I myself came across an archive page just the other day where a div issue was causing the dreaded "dropped sidebar" on my site.
I hadn't previously seen it, because I use Firefox as my primary browser.
Point is, how many people viewed that page in IE looked at that and went, "ugh, I cant read this" and left?
In the end, everyone is going to spend as much time as _they want_ honing their site and their skills. If you dont want a valid site, that dont do the necessary work -- and yes, it IS work. I, personally, find it to be very gratifying work, though.