I get asked that all the time as well and I simply explain to people that I'm Korean-American, son of immigrants. It's not that hard and it shouldn't be done defensively.

Also, that's a quote that my wife would love to hear once we have children.... ;)

This is why context and intent are everything. If we genuinely believe that the people saying and asking these things are doing it with the intention of angering or embarrassing us, then we live in a very sad world where cultures will never mix.