RE: Block.ops - An Analysis Tool
It occurs to me that this is exactly the sort of project that I would find extremely useful for a number of analyses that I would like to be doing but can't now that SteamData has gone the way of all flesh. I much prefer working with MongoDB than any of the alternatives currently accessible.
It also occurs to me that this project might profit substantially from real bifurcation, first focusing on a simple, straightforward way to pull content from the stream into a MongoDB and have that working first as an easily deployable standalone application, followed by analysis tools which speak to that MongoDB established by the first database.
For myself, I don't really need a set of tools implemented for me which can dig around in and analyze the database contents; I wrote a pile of Python code publicly a few months ago which would do that well enough, at least for my needs.
Getting blockchain contents into a database, on the other hand – that's more than slightly challenging. The method used by SteamData was open source and I appreciate that fact, but it was nontrivial.
This commit also includes the construction of various steem API functions for accessing AppBase. I have toyed with both Steem-js and dSteem but in the end I have built my own functions (from scratch and then using generic npm modules request and request-promise-native). There's no overwhelming reason for this choice, other than this being my first project of this type and I like to learn from the ground up.
Have you looked at beem? @holger80 has done an amazing job of building a very solid interface to the steem blockchain, accessible through Python. The Discord channel associated with the project is extremely well attended and I've seen some very insightful analysis of what are effectively some bugs in the methodologies implemented in the working specification of the blockchain itself come to light because of the way that he is working with unit tests and because of the projects that are already being made with beem.
Almost all days have some dropped / missed blocks in comparison to the expected number based on 3 second intervals. I'd be interested in understanding how this happens if anyone can point me to a good explanation. Generally the number is small but occasionally there are days with significant outages.
If anyone can answer this question, it's those guys.
I'm looking forward to seeing where the project goes, purely for my own self-interest, and I think there's a lot of potential for this to become very interesting.
Thanks @lextenebris, that's some thoughtful advice!
My own interest in carrying out the project is mainly for the production and automation of heavy / complex analyses that are difficult with the existing systems. A bespoke tool rather than an all-encompassing archive like steemsql or steemdata. That said, I do plan for it to be pretty flexible so that people can use it for different purposes. But there are serious volume considerations now with the full Steem data so I'm looking more along streamlined lines.
The coding does naturally split along those lines so it should be possible to separate off the first requirement as a standalone tool without difficulty. I don't expect it will take too long to build overall so hopefully that could be ready fairly soon. I will need to take a holistic approach to including the various parts as I build it though, simply because it's my first time using node.js, steem and Mongo so I'll be working out what works and what doesn't as I go along. But it will be modular, so splitting should be simple.
I have heard good things of beem! But I understand that it's Python-based so not something I can work with currently. Next year! I will check in with them on the dropped blocks though. Thanks for the hint!