Update to Technical Price Dashboard

Summary

I’ve made a number of key updates to the dashboard (located at http://35.229.125.96/ with username: Matt and password: m@tt!) in the last two weeks. If you would like review the previous update and original proposal, click on the links below:

• Original Proposal

• Update

The first update is that I’ve increased the number of currencies included in the dashboard from 25 to all Bitcoin markets on Bittrex (200+ currencies). The original proposal included 25 currencies. This means that the full order history for all Bitcoin markets is being saved to the database. The full list can be viewed in the bittrex.py file in the project Github repository (brophymj/bitgeek). The order history for all coins can be downloaded in the Reports Generator (screenshot below).

As before, you can enter the start and end date, and coin code to retrieve the order history for that period (see below). The file will be downloaded as a CSV.

Note: the additional coins have been populating the database since yesterday (08/02/2018). The 25 original currencies have been updating since the 23rd January. There are now over 2million records in the DB.

The graph feature is now also available for all 200+ coins. In addition, I spotted an issue with the MACD graph. The first EMA point in the graph should be set as a moving average of the last 26/12 points. In the original dashboard, the first EMA point was assuming the EMA for the previous period was 0 which was creating a distortion in the graph (a hump appearing at the start of the graphs).

Ripple (XRP) was one of the currencies added – I’ve included a screenshot of the MACD using 5-minute intervals below:

You can specify the interval of the analysis (1/5/10/30/60mins) and pick a historic end point for the graph (or just get the latest view). The Bearish/Bullish indicator will let you know if the price is trending upwards or downwards.

Addition of Data Centre Menu

This is the main change in this Update post. While it’s very useful to have the order history at the most detailed level, I found I was doing adjustments to the data before I could analyse it properly. After downloading the data, I would normally have to:

• Summarize the data into intervals of 1 min/ 5 min etc.

• Calculate a weighted average price for the period

• Fill in dummy rows where the data was missing. This might seems small but it was a headache to update! There is often data gaps (particularly for the lower volume coins) where there won’t be an order for 5/10 minutes.

• Download a separate file for each currency

I’ve addressed all of these issues in this update by adding the Data Center menu item, which looks like:

I’m generally interested in aggregated data i.e. analysing patterns across multiple coins. We can now download the data for all coins together by leaving the Coin Input as blank.

When you download data from the data centre, you can select the interval you want, the date range and a CSV file will be saved the Data Center page for 24 hours. It can take a while to download, so you can log out, get a cup of tea and log back in later to see your downloaded file. The screenshot below shows the file sitting in the Data Center page ready to be downloaded to your computer. In this case, I’ve downloaded data for all coins, using 10-minute intervals in the 24 hour period to 15:39 on the 9th February.

The file above has 27k rows and looks like this:

For the dummy rows, I assume the price is equal to the last traded price and the volume is 0. You will see the download also contains the MACD fields allowing you to graph the MACD (if you’re interested).

Key Code Updates

The code is all available in the GitHub repository. Below are extracts of the key additions:

The code below created dummy records where there are gaps in data.

pipeline =\
            [{"$match":
              {'TimeStamp':
               {"$lt":
                (datetime.strptime(todate,
                                   '%m/%d/%Y %I:%M %p'
                                   ) + timedelta(minutes=1)
                 ).strftime('%Y-%m-%dT%H:%M'),
                "$gte":
                datetime.strptime(fromdate,
                                  '%m/%d/%Y %I:%M %p'
                                  ).strftime('%Y-%m-%dT%H:%M')},
               'Pair': coin
               }
              },
             {"$group":
                {"_id":
                 {'datehours':
                  {"$arrayElemAt":
                   [
                       {"$split":
                        ["$TimeStamp",
                         ':'
                         ]
                        }, 0]
                   },
                  'minutes':
                  {"$arrayElemAt":
                   [
                       {"$split":
                        ["$TimeStamp", ':']
                        }, 1]
                   }
                  },
                 "sum_quantity":
                 {
                     "$sum":
                     "$Quantity"
                 },
                 "sum_total":
                 {
                     "$sum":
                     "$Total"
                 }
                 }
              },
             {"$project":
              {"_id": 1,
               "sum_quantity": 1,
               "sum_total": 1,
               "price":
               {"$divide":
                [
                    "$sum_total",
                    "$sum_quantity"
                ]
                }
               }
              },
             {"$sort": SON([("_id", -1)])
              }
             ]
 

The code below creates the CSV file that sits in the Data Center page for 24 hours:

def datacenter_report(interval, todate, coin, fast, slow, signal, fromdate):
    """Generate report and CSV file for datacenter endpoint."""
    if coin:
        path = [timing(fromdate), timing(todate), coin, str(interval)]
    else:
        path = [timing(fromdate), timing(todate), 'ALL', str(interval)]
    filepath = '-'.join(path).replace(':', '-')
    with open('archive/{}.csv'.format(filepath), 'w') as dump:
        fieldnames = ['pair', 'interval', 'datetime', 'date',
                      'time', 'volume',
                      'price', 'ema_fast', 'ema_slow', 'macd',
                      'signal_line', 'macd_hist']
        writer = csv.DictWriter(
            dump, fieldnames=fieldnames, extrasaction='ignore')
        writer.writeheader()
        if coin:
            [writer.writerow(i) for i in summarize(
                interval, todate, coin, fast, slow, signal, fromdate)]
        else:
            for c in coins_list:
                try:
                    [writer.writerow(i) for i in summarize(
                        interval, todate, c, fast, slow, signal, fromdate)]
                except:
                    pass
    return filepath
 

The code for the bug fix is:

        if not fromdate:
            generator = b[::-1][:interval * 67][::interval]
            summarized = generator[:40]
            ema_basic_slow =\
                sum([Decimal(i['price']) for i in generator[41:]])\
                / Decimal(len(generator[41:]))
            ema_basic_fast =\
                sum([Decimal(i['price']) for i in generator[41:53]])\
                / Decimal(len(generator[41:53]))
        else:
 

Benefits/Uses of this Update

This project started as a personal interest of mine, which is modelling future cryptocurrency prices. The volatility and irrationality of crypo markets creates (in my opinion) opportunities to gain from buying and selling at the rights times. I’m interested in whether huge intra-day spikes are predictive and if there are signals which can be generated that will inform you of a likely upcoming changes in the direction of the price.

So, the main use is data analysis and modelling. You don’t have to be a rocket scientist to do this, you can download the data, create a few graphs and inspect for yourself. You might see something that looks like a predictor and build a model around that predictor and test it.

If I could summarise this phase of the project I would say building a toolkit to do the necessary data analysis. The next stage is modelling and actually parameterising a model that will let you know when (according to your model) to get in or out.

Next Steps

For the next update, I’d like to incorporate additional graphs that will also feed into the data downloads and help with the analysis part. The graphs I intend to add in the next update are:

• Relative Strength Index

• Bollinger Bands

How to get involved

As with the previous posts, I’d love to hear your feedback and thoughts on this. In particular, if you had insights into approaches to analysing the data, I’d be really interested in hearing them.

Also, some feedback on the updates (good and bad) would be very welcome.

Many thanks for your attention.

---

Picture Source: Pixabay.com

---

Posted on Utopian.io - Rewarding Open Source Contributors

---