Overbooking is a simple case of the free market. Back in the 50s, 60s airlines prided themselves on offering premium services, and the businesses paid for it. Then came low-cost airlines and the consumers spoke overwhelmingly, there was only one thing they were looking for - cheap prices. Airlines had to scramble to cut costs drastically, and overbooking was a clear low hanging fruit. Their algorithms have gotten pretty good at it too - I believe United denies boarding to less than 20 out of 10,000 passengers. Of course, it is in their terms of service, and the consumers have spoken loudly about choosing low prices overall given the low risk of being denied boarding.

This is why government owned/subsidized airlines like Emirates or Singapore Airlines offer a far superior service to any US airline which receives far fewer subsidies - they can afford to stay away from the rat race and focus on quality of service. That said, certain airlines do it better than others - I think JetBlue has a far lower rate of denying boarding than other US airlines. Yet, consumers will continue to opt for saving pennies, and overbooking will continue.