I have very similar stories in my family and yet none of my grandparents (as far as I know) had any involvement with the occult. I was the only one in my family who studied and practiced the occult.
It wouldn't be fair in my mind that my descendants should have to suffer for something I have done. If God wills it, I will endure whatever punishment I have coming for my sins, but I will do all I can to teach my children the bible so that they don't go astray as I did.

I believe that God does not punish innocents. And I understand the word used in Exodus and Deuteronomy is "paqad" which is translated as "visiting" in KJV and sometimes as "punishing" and "appointing" but it has the connotation of checking, caring, observing. I believe God keeps score of each and every of our action and choice, and idol-worship is indeed the greatest sin one can commit. So, we need to be extremely vigilant when it comes to that because if most of our family tree are idolaters, you're almost guaranteed to expect tragedy if you continue with tradition (and conflict if you break from tradition).