Fceux remote controlling

Repository

https://github.com/spiiin/fceux_luaserver

What is the project about?

In short - it's a script, that makes NES emulator remote-controlled, and server, that can send commands to the emulator.

This project is also related to my CadEditor project - Universal Level Editor project, and powerful tools, needed to explore games, so I also used my fundition.io project tag.

Why someone needs it?

Several emulators, and Fceux also, can run Lua-scripts for controlling them. But Lua is a bad language for general programming. Basically, it's the simple language for calling C-functions. Authors of emulators use it as a scripting language only for one reason - it's lightweight. Accuracy emulation needs many CPU resources, and scripting is not the primary goal for authors.

But now, personal computers have enough resources for NES emulation, so why not used powerful scripting languages like Python or JavaScript for write script for emulators?

Unfortunately, there are no mainstream NES emulators, that can be controlled with these languages. I know only about Nintaco, it's also has fceux core, rewritten to Java. So I want to create my own project.

It's a proof-of-concept only - it not robust, or fast, but it's working. I created it for myself, but there is the frequent question, how to control emulator externally, so I decided to publish it's sources as is.

How it works

Lua side

Fceux emulator already has several Lua libraries embedded in it. One of them - LuaSocket library. It's not a lot of documentation about it, but I found code snippet in XKeeper Lua-scripts collection. It used sockets to send commands from Mirc to fceux.
Actually, code that create socket:

function connect(address, port, laddress, lport)
    local sock, err = socket.tcp()
    if not sock then return nil, err end
    if laddress then
        local res, err = sock:bind(laddress, lport, -1)
        if not res then return nil, err end
    end
    local res, err = sock:connect(address, port)
    if not res then return nil, err end
    return sock
end

sock2, err2 = connect("127.0.0.1", 81)
sock2:settimeout(0) --it's our socket object
print("Connected", sock2, err2)

It's a low-level tcp-socket, that send and receive data per 1 byte.

Fceux lua main cycle looks like:

function main()
    while true do                --loops forever
        passiveUpdate()     --do our updates
        emu.frameadvance()    --return control to the emulator, and it's render next frame
    end
end

And update cycle looks like:

function passiveUpdate()
    local message, err, part = sock2:receive("*all")
    if not message then
        message = part
    end
    if message and string.len(message)>0 then
        --print(message)
        local recCommand = json.decode(message)
        table.insert(commandsQueue, recCommand)
        coroutine.resume(parseCommandCoroutine)
    end
end

It's not very hard - we read data from the socket, and if next command in it, so we parse and execute it. Parsing made with coroutines - it's powerful lua concept for pause and resume code execution.

One more thing about fceux Lua system. Execution of emulation process can be paused from Lua-script, how it can be resumed from the socket, if the main cycle is paused?
Answer - there is the one undocumented Lua function, that will be called even if emulation paused:

gui.register(passiveUpdate) --undocumented. this function will call even if emulator paused

So, we can pause and continue execution with it - it will be used for setting up breakpoints from a remote server.

Python side
I created a very simple RPC-protocol, based on JSON commands. Python serialiazed command and arguments to JSON string and sent it via socket. Next, it wait to answer from lua. Answers has name "FUNCITON_finished" and field for results.

This idea encapsulated in syncCall class:

class syncCall:
    @classmethod
    def waitUntil(cls, messageName):
        """cycle for reading data from socket until needed message was read from it. All other messages will added in message queue"""
        while True:
            cmd = messages.parseMessages(asyncCall.waitAnswer(), [messageName])
            #print(cmd)
            if cmd != None:
                if len(cmd)>1:
                    return cmd[1]
                return
               
    @classmethod
    def call(cls, *params):
        """wrapper for sending [functionName, [param1, param2, ...]] to socket and wait until client return [functionName_finished, [result1,...]] answer"""
        sender.send(*params)
        funcName = params[0]
        return syncCall.waitUntil(funcName + "_finished")

So, with this class lua methods can be encapsulated in Python classes-wrappers:

class emu:
    @classmethod
    def poweron(cls):
        return syncCall.call("emu.poweron")
        
    @classmethod
    def pause(cls):
        return syncCall.call("emu.pause")
        
    @classmethod
    def unpause(cls):
        return syncCall.call("emu.unpause")
        
    @classmethod
    def message(cls, str):
        return syncCall.call("emu.message", str)
        
    @classmethod
    def softreset(cls):
        return syncCall.call("emu.softreset")
        
    @classmethod
    def speedmode(cls, str):
        return syncCall.call("emu.speedmode", str)

And called exactly how it called from lua:

#Restart game:
emu.poweron()

Callbacks
Lua can register callbacks - functions, that will be called after certain conditions. We can encapsulate this behavior in Python. It needs some additional trick toimplement it.
At first, we save callback function handler in python, and save this callback to lua.

class callbacks:
    functions = {}
    
    callbackList = [
        "emu.registerbefore_callback",
        "emu.registerafter_callback",
        "memory.registerexecute_callback",
        "memory.registerwrite_callback",
    ]
    
    @classmethod
    def registerfunction(cls, func):
        if func == None:
            return 0
        hfunc = hash(func)
        callbacks.functions[hfunc] = func
        return hfunc
        
    @classmethod 
    def error(cls, e):
        emu.message("Python error: " + str(e))
    
    @classmethod
    def checkAllCallbacks(cls, cmd):
        #print("check:", cmd)
        for callbackName in callbacks.callbackList:
            if cmd[0] == callbackName:
                hfunc = cmd[1]
                #print("hfunc:", hfunc)
                func = callbacks.functions.get(hfunc)
                #print("func:", func)
                if func:
                    try:
                        func(*cmd[2:]) #skip function name and function hash and save others arguments
                    except Exception as e:
                        callbacks.error(e)
                        pass
                    #TODO: thread locking
                    sender.send(callbackName + "_finished")

Lua server save this handler, and call generic python callback function with this handler.
After that, in Python, we create additional thread, that checks, if registered callback function need to be called:

def callbacksThread():
    cycle = 0
    while True:
        cycle += 1
        try:
            cmd = messages.parseMessages(asyncCall.waitAnswer(), callbacks.callbackList)
            if cmd:
                #print("Callback received:", cmd)
                callbacks.checkAllCallbacks(cmd)
            pass
        except socket.timeout:
            pass
        time.sleep(0.001)

Last step - callback function do some work and return flow control to lua caller.

How to run example

• You must have working Python and Jupyter Notebook in your system. Run jupyter with command:

jupyter notebook

• Open FceuxPythonServer.py.ipynb notebook and run first cell:
  

• Now you must run fceux emulator with ROM (I used Castlevania (U) (PRG0) [!].nes for my examples).

Next, start lua script fceux_listener.lua. It must connect to running jupyter python server.
I do all these things with one command-line command:

fceux.exe -lua fceux_listener.lua "Castlevania (U) (PRG0) [!].nes"

• Now go back to Jupyter Notebook and you must see the message about the successful connection:

You are able to send commands from Jupyter to Fceux (you can execute Notebook cells one by one and see results).

Full example can be viewed on github:
https://github.com/spiiin/fceux_luaserver/blob/master/FceuxPythonServer.py.ipynb

It contains simple functions:

Callbacks:

And even complex script for moving Super Mario Bros enemies with mouse:

Video example

Limitations and applications

The script is not very robust, it needs additional checks for all data, received from the network. Also, it not very fast. It's better to use some binary RPC protocol instead of text. But my implementation not need compilation, so it can be used "as is". The script can switch execution context from emulator to server and back 500-1000 times per second. It's enough for almost all applications, excepts some special debugging case (for example per pixels or per scanlines PPU debugging, but Fceux does not support it whatever).

Other possible applications:

• example for other remote-controlled emulators
• dynamic reverse-engineering for games
• add cheating or tool-assisted superplay abilities
• injecting or exctracting data or code to/from games
• extending emulator features - create 3rd party debuggers, recording replay tools, scripting libraries, game editors
• netplay, control emulator via mobile devices/remote services/remote joypads, cloud saves/patches
• cross emulator features
• using python's or other languages' libraries to analyze games' data or to control game (AI-bots)

Technology Stack

I used:
Fceux - http://www.fceux.com/web/home.html
It the classic NES emulator, that many peoples using. It not updated by a long time, and not the best by features, but it still defaults emulator for most romhackers. Also, I chose it, because it already includes several Lua libraries, LuaSocket is one of them, so I don't need to implement sockets myself.

Json.lua - https://github.com/spiiin/json.lua
It's pure lua json implementation. I used it because I want to provide the sample, that no need compilation). But I need to create the fork of the library because some other lua library inside fceux code (there are no sources for these libraries) override lua system tostring function, and it brake json serialization (my rejected pull request ) to the original library.

Python 3 - https://www.python.org/
Fceux Lua script tries to open tcp-socket and to listen for commands sent via it. Server, that will send commands to the emulator, can be implemented with any language. I used python because of it's the philosophy of "Battery included" - a lot of modules included by default (socket and json also), and others can be added without problems. Also, Python has libraries for working with Deep Learning, and I want to experiment with it for creating AI for NES games.

Jupyter Notebook - https://jupyter.org/
Jupyter Notebook is cool python environment, with it, you can write python commands (and not only) in table-like editor inside browser interactively. Also, it's good to creating interactive examples.

Also, I used dexpot, because I need to pin fceux window topmost. Windows can't do that by default, and that window manager can. Also, it is free for personal use.

Roadmap

This is proof-of-concept of the remote controlling emulator. It can be used as a base for other projects. I implemented all Lua functions, that emulator has. Other more complex examples can be implemented and committed to the repository. Several improvements for speed can be done also.

Links and similar projects

Nintaco - Java NES emulator with api for remote controlling
Xkeeper0 emu-lua collection - collection of many Lua scripts
Mesen - C# NES emulator with powerful Lua script abilities (with no embedded socket support for now)
CadEditor - Universal Level Editor and powerful tools for exploring games. I used the project from post to explore games and add it to the CadEditor.

How to contribute?

Use it, test it, explore NES games with it. Send me your scripts by pull requests.

My github account

https://github.com/spiiin