I have drawn up some sketches of a 20'x20' smallish house that would allow me room for my hobbies, library, storage, and general living space. But were I to add a family, it would be too cramped by far. Some would call it too small, others would say it's extravagant. But it would suit my needs, and that's what matters.
The hardest part of architectural design is separating needs from preconceptions in order to serve the specific needs of the client.