Speakers absolutely work as inefficient microphones, and there is no reason that a microphone can't be made using a magnet/inductor combo. In fact I'm sure many of these exist. If I had to guess why electret microphones were more common, I would guess that they are more compact and less prone to mechanical failure, but I'm not actually sure.