As a coin developer and also maintainer of a mining pool, I can honestly say even large drops are totally normal... More there is hash rate, more drastic drops are normal as stratum difficulty depends totally on individual hashing powers of miners and stratum difficulty, not block difficulty, is what dictates when individual miners submit their hashes...

To sum things up, there will always be large drops, because the network eventually needs to stabilize, otherwise time between consecutive blocks would keep increasing until no more blocks can be solved anymore. Average block difficulty will always drift away from the target difficulty, but it will never drift away far enough to cause network stall...