Good read, Derrick! The biggest issue I see around this topic is this:

A lot of libertarians mix up their principles and their personal preferences. You've already said it: it shouldn't be a problem if like-minded people want to live in close proximity, and there are benefits to it, but that doesn't mean that diversity is necessarily dangerous.

Often we see that the result is that libertarians want to impose 'their form of liberty' on others. They go back to the statist way of doing things (look at Chase Rachels and his closed borders advocacy).