I agree that is a social construct that we individually create and the ideal would be that each person can live and express their concept of gender. If there were either no people around or if general society didn't care how a person dressed or expressed themselves, I don't think we would even have a word for it, let alone a problem.

Unfortunately interacting with others in society is necessary and there are real pressures either to conform and/or reactions for towards those that aren't conforming. Even when direct confrontation or violence aren't involved, the stares, the whispers, or uncomfortable tensions from others are very real.