Its perfectly possible. In fact, I'd think that was one of the objectives of XAMPP.
How to set it up?
Just plug your USB HDD into the PC you want to, then run "setup_xampp.bat" from the root folder of XAMPP installed directory. That will setup all the paths in all the config files. Please note that the most recent release is having some problems with the paths re-configuration, as I have mentioned here - http://community.apachefriends.org/f/viewtopic.php?t=10762