Repository

https://github.com/imwatsi/trader-bfx-backend

Backend code for a trading assistant bot (Bitfinex)

---

The major highlight of this update is the Settings module, which enables customization. Previously, it wasn't easy to get the bot to handle many markets at the same time. Now, the bot is market-agnostic, and things like candles, order books, tickers, and later on; trading, analysis, alerts, etc, can be done on multiple markets concurrently, thanks to the settings module.

I also added other features on the roadmap: order books and tickers support.

1. Order Books

Commit: subscribe to order book

This commit adds code to handle order book subscriptions as well as capture their channel information.

Commit: parse order book

The code above ignores heartbeat messages coming from order book channels and directs channel data to import_books() function in t_market.py, which handles the parsing and storage.

• A quick check for the market key is performed before everything else, adding the market to the dictionary if it's not there already.
• Snapshots and update data are differentiated by the type of data field: snapshots are lists and updates are singlular.

• A nested function, parse_book_entry(), converts individual entries into workable dictionaries.
• A second nested function, update_book(), uses the individual entry to update order books: either adding a price level, updating it or deleting it (according to Bitfinex API conventions)

2. Settings

The bot now has a settings module, which handles the storage and extraction of settings. I used a settings.json file for storage.

Design

The settings file will be the base on which the UI will be built and its design had to incorporate future UI plans.

• Global settings enable and disable system wide features. Trading markets can be added and removed, resource intensive functions like order books and tickers that provide realtime streams of data can be disabled completely.
• Individual markets can have multiple candle-timeframes, allowing analysis to be performed on multiple markets and multiple timeframes, all concurrently
• There is room to have candle-timeframe-specific settings for the various technical analysis tools that will be implemented later.
• Order book subscription is now market-agnostic. Each market can have its own order book settings like length, precision and frequency. It's also possible to disable the order books feature on a per-market basis.
• Tickers are also market-agnostic, and they can be enabled/disabled individually or collectively.

Ineffective attempt

Before using JSON as the file format to store settings, I first attempted to use .ini format, with configparser. That proved highly ineffective. The two commits below show how I had implemented it:

• Commit: add settings module

• Commit: integrate and expand settings

Commit: use json for settings

This commit added JSON formatting for settings. It dealt with a number of aspects, I'll break it down:

Settings class

In the screenshot above is a class I created to avail a settings object globally, hosting variables that contain all settings. Settings are taken directly from settings.json.

• if __name__ == "t_settings": handles the first instance of loading
• refresh_settings() handles the periodic reloading of the settings file, to implement new configuration changes made by external apps (i.e. the UI).
• The function refresh_settings() is called from t_init.py, and since it's a perpetual loop, it's called from a new thread.

This commit also integrates settings features in the logic of the bot itself, making it responsive to different configurations. Appropriate modifications were made to the t_api_socket.py script.

The rabit hole

As I worked on this module and performed tests, I noticed that a corrupted settings file would break the system completely. To avoid this, I ended up going down a rabbit hole, proactively creating processes to check the integrity of the settings file itself. More work still needs to be done on this front.

This was also a good chance to implement a feature to repopulate a missing settings.json file.

Integrity check function

The above process resulted in the creation of check_integrity(), a function that helps detect and correct invalid/duplicate entries in the settings file. It also repopulates a new settings.json file if it's missing.

• check_missing_settings() checks for missing entries in the "bot-wide settings" section, and market specific sections like candles, order books and tickers.

• check_invalids() will be expanded to check individual data fields for incorrect formats and more.

3. Tickers

Commit: add tickers feature

• To add the tickers feature, I wrote code to subscribe to the API channel (per-market, toggled on/off by settings):

• Then I added code to handle the messages as they come from the API, redirecting data to import_tickers() in t_market.py:

• After that, I added a function to parse and store the tickers in memory:

4. Miscellaneous improvements

Before implementing most features in this update, I moved code responsible for parsing candle data to the t_market.py module for beter organization, with these two commits:

• move candle handler
• move candle handler 2

I also made a few changes to improve code readability:

• minor improvements

5. Data samples (Market data ready for manipulation)

Below are screenshots of data contained in the three t_market.py variables during runtime. The dictionaries are ready to be manipulated for analysis, etc, which is what I'll be working on next.

Candles: market, timeframe, candle {open, close, high, low, vol...}

Books: market, asks/bids, price {amount, count}

Tickers: market {ask, ask size, bid, bid size, daily change, daily change %, high...}

---

That's all for this update. I'm now working on technical analysis implementations.