For the moment, the score is just something funny. Like you said, a kind of badge of merit. This is by no means affecting any future vote as the rules for votes are independent (and clearly stated).

The decaying reputation is more to have a way to slowly remove users that have left. There is a difference between not being present and having left, which is why the half-life is large. However, this may be improved (cf. the above comment by @tarazkp).