How I made my own python bot to automate complex games

This may seem like a simple task but it's really not. I spent hours and hours on this program. And I'd like to share my experience and what I've learned with you.

The game

First what is it about ? the game is a 2d mmorpg, Wakfu http://wakfu.com/. I invite you to look it up, it's honestly a great game, it's a french game but it has an international server as well.

There are several mechanics in game. I'll talk mainly about how I automated fights because I spent most of my time automating that game mechanic.

Fighting

The fight system is very close to final fantasy advance, you can have up to 6 players in a fight versus monsters.

For those of you who don't know final fantasy tactics, it's pretty simple :

You have a grid where both monsters and players are placed. They can move a fixed number of squares per turn, and have spells that have various effects (heal, damage, apply effects like stun etc) which have various range and area of effect.

Here you can fight a short video of a guy doing a dungeon (so several fights) so you can see how it looks :

(ignore the french commentary)

Also note that each turn is 30 seconds so you can't take too much time to compute.

Anti bot protection :

There is also a protection against abuse. Which took some time to beat as you can imagine. introducing, the capchat :

You have your character and a cat in front of you, around the cat you see three squares with numbers (ranging from 1 to 8). Around you you have 8 squares with every numbers.

You need to click on the squares around you that matches the squares around the cat.

This is a relatively easy task right ? simply run an image search algorithm and boom you're done.

Yes except :

• The grid tilted which means that you have to create squares out of diamonds (which means smaller images to compare and so more imprecisions). Unless you do an imagesearch which handles alpha channels but this adds lots of complexity and see point 2.

• The numbers are constantly moving. See this small video :
  

(actual protection starts at 1:13)

This is a 2 :

This is also a 2 taken a few second after :

And this is a 3 :

See where I'm going ? since the numbers are constantly moving and the difference is very subtle it's almost impossible to classify them using traditional algorithms.

Automation

So we have a pretty complex game, how are we going to automate that ?

first we have to choose a language, I usually code bots in Autoit. It's a GREAT language for automation, in fact it was designed for it. But I wanted my bot to be cross platform so I opted for python (3 obviously, for the love of god stop starting new project in python 2.7). because it's pretty versatile, has some machine learning libraries (we'll come to that later) and I find it pleasing to code with it.

It will take a long time to talk about my whole experience and so I will split that into several articles for an easier reading experience. this will be the summary of all the steps taken. Most of the following articles will be written to be kind of a tutorial or ressources so future readers can find value in the articles and find them on google.

step 1 : Controlling the mouse and keyboard with python.

step 2 : Image recognition with python.

step 3 : How to create your own dataset for machine learning and tensorflow

step 4 : Computer vision and machine learning via tensorflow.

Step 5 : How to save and reuse a trained model in tensorflow.

step 6 : How to time your code and benchmark via python.

step 7 : How to automate a game with random bugs.

I may add more later so follow me to get updated :)