I guess I'm just a hard nose on these things.

I'd turn them away regardless and write them a polite letter telling them that there seems to often be confusion about what was said and done when speaking in person and that you want all conversation with them to be in written form via postal mail from now on so that everyone knows exactly what was and was not said.

I'd also let them know that they are no longer welcome at your home and you will not let them in going forward without a warrant.

Don't let them scare you. Make them make a record when speaking with you.

If they say "You used to let us in!" let them know that you did that before you realized that they were a bunch of child stealing liars.