By distribute, do you mean distribute other complete installations or just distribute stuff that can be uploaded and installed elsewhere?
If you want it to be totally portable and easy to distribute, you'd need to find a portable version of MySQL, PHP, Apache, etc. and load them on to the thumb drive. Then, you could just install and configure your WordPress installation like you'd expect and it would be relatively easy to do.
If you want it to be a little bit less out of the box, you could set it all up, zip the files, and include a database export. The new user would then need to import the database and upload the files and would be ready to go.
Hopefully that helps.