Hmm, for any native speaker, the distinction is clear. That's probably because we all know the "ER" suffix so well. Teacher, driver, seller, buyer, recorder, producer, runner, jumper, etc. So when we hear the "EE" suffix, we know it's the opposite.

For instance, we might joke that instead of being the "killer," Mr X is the "killee." ☺ Even though that word is not a word and is just nonsense, it makes complete sense, and we can all understand it.

Hope this helps you learn it.