Difference between tags and categories?

Basically, where does categories end and tags begin?

I like to keep as few categories as possible, to avoid having a messy blog, but then there’s suddenly tags thrown into the mix as well …?

What is the advantages of using tags as oppose to, or with, categories?