Strictly speaking, the amount of matter is not fixed. It can annihilate and give back radiation for instance. However, our standard knowledge breaks down when we approach the initial moment. In short, we don't understand what is going on. What we understand is that most implications of the model seem to be well explained by data.

By the way, there are alternative cosmologies. In some of them, in contrast to the standard model (the big bang model), there is no beginning and end. You may one to check Penrose's conformal cyclic cosmological model for instance.