I see where you are coming from, but even a small party, if voted into power, would impose their way of doing things onto the population by force and is therefore immoral.

The point I'm making is that anyone forcing anyone else (be it an individual or a government) to do something with threat of force is immoral and our society cannot progress when the very basis of our system is based on coercion.