I prefer soft bags over the hard.. most of what I travel with is either clothing or camping gear all of which usually goes into waterproof stuff sacks.. my leather bags have been between the bike and the ground a few times with the only damage being a few scuffs.. My brother has hard bags and I know he has had to completely replace them a few different times.
Yes, I've heard many similar stories. Plastic is possibly the worst choice as it has little to no resistance to impact and the scuffs are permanent. Leather you can polish up and even stitch if needed. Then there's that classic lived in look to leather that's kinda nice too.