The price is set by supply and demand. Mining difficulty follows the price, not the other way. In a scenario where a significant portion of the miners drop out of the network, the time between blocks (time to confirm a transaction) will increase. There is no direct relationship between the number of active miners and the price.