I guess it all boils down to competition. People want to build something bigger than what other people built. It's the way it goes. As to the material, it used to be that toxic things like asbestos were used to line structures. But now, I'm happy to see a lot of sustainable buildings that are not only durable but eco-friendly as well.
Well, I don't know if it's only competition. Here, people started building bigger houses because the circumstances were economically favorable and stable, so they have built the 1st floor for their children, and left it at the 4th phase - rough installations were built in. Besides they spent most time outdoors, either at work or farming. Now it's different, more like you say, except for sustainable. I'm not sure about that, at least here. Paints are certainly poison, even though they have a paint that functions as an insulator, as well. PasivDom company in Ukraine even prints house parts now. Personally, I prefer those little - sometimes even self made - houses or hoses made out of straw bales - but this can only be done if the area building plans permit.