How to control the mouse and keyboard with python for automation
How to control the mouse and keyboard with python
This is an article that is part of a serie that I initiated regarding my bot. You can find more about it here
The first step to automate anything is, of course to be able to replace the basic Human to machine inputs.
For 99% of applications it is simply the mouse and keyboard. So what do we need ?
Being able to input anything with the keyboard :
Keys
Shortcuts (combinaison of keys)
Being able to input anything with the mouse :
Click right, middle, left
Move anywhere
And all of that working in a cross platform (by cross platform I mean windows/linux, I don't really care about macOs users since I don't own one and linux-compatible scripts tends to work on macos as well)
Now after some research I found the library that I was looking for : pyautogui
Pyautogui
Pyautogui is a library that allows you to control the mouse and keyboard to do various things.
On windows the installation is a simple as
pip install pyautogui
(if you have pip in your PATH of course), if not you need to find "pip.exe" in the folder where you installed python.
On linux you need to install a few tools before being able to install it :
pip3 install python3-xlib
sudo apt-get install scrot
sudo apt-get install python3-tk
sudo apt-get install python3-dev
pip3 install pyautogui
They have a GREAT documentation that you can find at https://pyautogui.readthedocs.io
If you want to start experimenting with it fast, then you should look at the cheat sheet
Keyboard
I was looking for a way to control the keyboard :
To type one (or several) keys I used
pyautogui.press(keys)
To do some more specifics actions that needed some timing I used
pyautogui.keyDown(key_name)
pyautogui.keyUp(key_name)
In order to do things like a control + click then I do keyDown + click + keyUp
Mouse
Controlling the mouse is as easy :
pyautogui.moveTo(x,y,duration=num_seconds)
basically "move to position x y in z seconds"
Note that in an usual screen like 1080p we move from one end to the other in less than a second.
I usually put 0.2 to 0.4 seconds.
To click I usually do a "move" and then a "click" because I ran into some problems where my game would not like if things go to fast, so I do that this way so I can put a little sleep inbetween. But it's totally fine to directly call the click function with all the args
pyautogui.click(button=button)
or
pyautogui.click(x=moveToX, y=moveToY, clicks=num_of_clicks, interval=secs_between_clicks, button='left')
and that's about it, the libary allows you to do a lot of things while giving you a lot of control. It's definitely my goto library for any automation in python now.
Thanks for reading and stay tuned for my next subject : Image recognition with python.
Good idea for the post.
I am a programmer myself and what I would like to see is the full script in a Github gist or somewhere else. That would make it easier to comprehend.
Anyway you get an upvote! Programming topics are fun :) keep going!
Thank you for the tip ! I will try to add snippets since the whole script is quite long and it would be confusing.