Sometimes it is hard though to tell if a user is actually writing a spam comment or if English just isn't their first language and they are struggling.

Yes, that happens to me too sometimes but then I check the profile and comments from the user to see I am not making a mistake or something ;)