Why I had to use machine learning for bypassing the anti-bot security

Hello,

This is the fourth part of a serie "automating games with python"

You can find on my account How I made my own python bot to automate complex games (part 1) Which explains my motivation and the game I'm automating itself. the part 2 is How to control the mouse and keyboard with python for automation Which digs into the core functions that are needed for automation. Part 3 which talks about a wrapper that I made to easily implement image searching within your python program and how to use it : Image recognition with python

Now, why there is a need for neural net-powered image recognition when we already have a library that can do that ?

I mean, when you think about it machine learning is really a heavy duty machinery :

• You need a big amount of image samples

• Training can take a lot of time

• Using machine learning is not trivial at all

Well, it's simple : precision on hard to find images

By hard to classify I mean images that can move.

For instance, how do you detect a character that is constantly moving in its idle position (a street fighter character for instance), or images that change color. If you are gonna try to find that using typical image recognition you need to find a picture that is constantly changing, so you have 2 choices :

• Take many different pictures of the different states of the moving picture and stack imagesearch onto imagesearch to try to detect it. But it's very slow and still pretty inaccurate.
• Take one picture and set a high tolerance for pixel changes, this can bring up lots of false positives, so it's still bad.

So this can be solved pretty easily with machine learning, where yous simply input a lot of images and the algorithm learns itself how to properly detect the images with an almost perfect accuracy in most cases.

Back to our problem :

What do we need to classify ?

If you remember my first post, You'll remember about the anti-bot protection, if you don't here is the original text :

Introducing, the capchat :
You have your character and a cat in front of you, around the cat you see three squares with numbers (ranging from 1 to 8). Around you you have 8 squares with every numbers.

You need to click on the squares around you that matches the squares around the cat. And note that the dots that makes the number inside a square constantly move :

This is a 2 :

This is also a 2 taken a few second after :

And this is a 3 :

See where I'm going ? since the numbers are constantly moving and the difference is very subtle it's almost impossible to classify them using traditional algorithms.

Before resorting to machine learning, I tried several things :

• Counting the dots : I took an image of a single dot and tried match how many were in the square, This did not work because they sometime blended and so it did not reach a good accuracy
• The other techniques described in this post (cascade and high tolerance), this does not work because it's not accurate

So I was up against the wall, The only solution that would work for me : Machine learning

I'm sorry for delaying by yet another post the actual machine learning things, but this post was supposed to be the intro and it's already way too long. And I don't want my articles to be too long, so we will see data preprocessing in the next article !

Thanks for reading :)