I agree with you.
For me, the problem is not with legal responsibility but with moral.
Politicians will always promise some measures and eventually act in a different way, so don't think you can be legally responsible for what they actually do, but just for what they promise to do.
Going to vote means that you are giving moral and political legitimacy to the whole representative system.