So how does an author straddle that line between Mary Sue and Torture Pinata?

I actually don't know.... But try to make the characters personalities from people personalities from real life, There's no Mary Sue/Stu in real life... every real person has flaws. (I think Superman isn't a good mary sue example, at least half of time, his personality is flawed depening on the writer).

How do you give them challenges to overcome without making them fly?

Probably it's good to make them overcome them sometimes and lose other times, and try to make them lose because they can't win... Mary Sue is usually isn't the character fault but because the writer can't use the character with good Narrative that they seem unstoppable.

That's why I said Superman isn't a good Mary Sue example.... His always winning is believable (depending on writer), his personality is flawed (we're not JUST informed that his personality is flawed) and most of all the Narrative isn't making him out a better chracter than he is.

at least that's the Superman I watched as a child, which Superman version are you talking about? Mary Sue depends much on how the story is Narrated than the actual story that happens.