If you go, they win. don't let the noise of others' opinions drown out your own inner voice. You are probably one of the best TA's out there. The trolls make you reach inside yourself so they are actually doing you a favor in a kind of way. Problems make you grow and in that way your adversity will become your advantage. I have experienced this first hand.