Math is nothing more than a method of describing our physical world. We made it up. Sometimes our made up math leads to artifacts that cannot exist in the read world. For example the number i that is the square root of -1. It is not real, and cannot be real and we flat out call it an imaginary number, but it works as a crutch to get our math past some roadblocks.

Just because math is a human creation does not mean it is arbitrary or illogical, it just means that humans have spent a long time perfecting it. It is sort of like a language. As time goes on, it evolves from simple grunts to an elaborate language used to convey extremely complex ideas. Languages are human creation. Math is merely another language.

Why does it matter? Well for one thing realizing that math is something we created to describe the world, means that math does not necessarily define the world. Just because we could potentially model alternate dimensions and travelling backwards in time mathematically doesn't mean it is actually a reality. It could just be an artifact of making the math work, just like the square root of -1 is imaginary.