It's okay. I am often heard telling my kids that I don't want "sorry" for this, that and t'other - for exactly the reasons you mentioned. The words have become cheapened. We (society) say "sorry" so frequently for meaningless (or disingenuous) things that it's easy to lose the words completely.