Fine-Tuning

Why is a parameter that is very close to zero strange or considered a problem? Isn't zero just a number like all others and we are not worried if, for example, a number is very close to 34.

From a bayesian point of view using a flat prior this is indeed the case. In a log-prior very small numbers may even be favoured.
Essentially, a flat prior means that you randomly select a value in the permitted range and each value is equally likely. 0.001 is as likely as 2.001 or 2.471. But there may be a few subtleties to consider:

• technical naturalness: If a parameter is small, but not protected from loop-corrections, then there is a real problem

• if the region close to zero is dynamically distinct and is therefore squeezed also in other parameters. In that case the bayesian evidence for this model will be rather small. It in fact does not depend on the parameter being small, but often this limit will provide a significantly different dynamics

• is there a model with higher significance? If a parameter is very close to zero, chances are high that there is another theory with more symmetry and less parameters. Then this theory will usually have a much higher significance. In that sense a parameter being very small can be a sign of a theory with too much freedom. This does not have to be the case and it could also be the case for a parameter that is not zero, but more often than not, this is a sign for the existence of a simpler more fundamental theory.