Block.ops - An Analysis Tool - Steem Currencies and Testing

Some hectic days of coding, trying to get this project up and running for some month end analyses. I'm running a little late!

This contribution mainly covers:

• Implementation of Steem currencies and their exhange rates:
  The blockchain data is expressed in many "currencies" (Steem, SP, Vests, SBD, STU, rshares) but most analyses will want results in a single currency, typically STU (what we see on post payout and vote amounts) or Steem. This new feature provides exhange rates between the various currencies.

• Testing / validation to ensure data loaded is complete and robust.
  Testing of my first batches of loaded data revealed a number of issues. This feature adds a number of tests and validations to ensure the loaded data is complete and robust. I also enhanced the active_votes logging process as this was the main cause of the issues.

In addition I added a number of other new features (mainly smaller features):

• App name / version processing improvements.
• Post/comment split.
• Improvements to the process of logging errors.
• Converting string dates to timestamps
• Additional self-validation for original posts vs edited posts.

---

For the last year I have carried out a monthly analysis of the Steem blockchain activity by application (i.e. by the different websites and applications through which you can post to the Steem blockchain - often termed dApps).

My aim is now to build a tool that can automate such complex analyses of Steem data, providing both historic time series and rapidly updatable real-time results. In addition to the dApp analysis there are many other projects for which such a system could be useful.

This tool is Block.ops.

You can read all about block.ops (including the project aims, technology stack, roadmap and how to contribute) in the introductory post here:
https://steemit.com/utopian-io/@miniature-tiger/block-ops-an-analysis-tool-1537300276791

---

Repository

https://github.com/miniature-tiger/block.ops

---

New Features

Implementation of Steem currencies and their exhange rates

The "currencies" of the Steem economy are complex!

The blockchain data is expressed in many currencies. Payouts for authors, curators and benefactors are recorded in a mixture of SBD, Steem and Vests (for Steem Power). The vote data uses rshares as the metric for vote size. The comment data and user interfaces on dApps use STU (i.e. SBD at $1 rather than market price) to illustrate vote sizes and post payouts.

Most analyses will want the results in one currency. Typically this would be STU since this is the measure most commonly displayed by the Steem dApps. However the blockchain API give no access to historic fx rates between these different measures.

This new feature solves this problem, providing exchange rates between the various currencies, allowing the results of analyses to be presented in a user-friendly format.

There are a number of options that could have been used to achieve this:

• Use the data to implement the full workings of the reward pool, the 3.5 day median witness feed averaging, and the steem price inflation. A project for another day!
• Extract the data from every post and comment in addition to every block operation so that the payouts are available in all currencies. Pretty inefficient for historic analyses.

Instead I have implemented an "empirical" solution. This chooses a single suitable post for each hour (according to various criteria), pulls the data for that single post and uses the payout data to derive fx rates. For example the curator payout in STU from the post can be compared to the total vests from the curator_reward payouts from the virtual block operations to give a Vests/STU fx rate for each hour. A similar approach is applied for Steem and rshares fx rates.

This approach allows a table of hourly fx rates to be built which can then be applied to all payouts.

These charts show the results of this process for data between 1 September and 11 September 2018. All rates are against 1STU (so may show the inverse of what you are used to seeing).

There are a couple of blips on the charts which require investigation but otherwise the results looks sufficiently stable for use.

The code changes are here:
https://github.com/miniature-tiger/block.ops/commit/4eb3eeb2b5fcb0de51b18a4691b8dc87841d9a84

Testing / validation to ensure data loaded is complete and robust

Testing of my first batches of loaded data revealed a number of issues. This feature adds a number of tests and validations to ensure the loaded data is complete and robust.

Numerous checks were implemeneted or upgraded to validate the cohesiveness of the data. Including:

• Report Blocks Processed: Upgraded to compare the blocks processed on each day against the total blocks that exist for that day, as well as the breakdown between ok, processing, and error statuses. As an example:

• Block Check Total: Check total on each block to validate all operations have been processed and uploaded (with allowance for skipped operations).
• Check Total of active vote processing: Active votes processed and logged vs active votes on the comment.
• New function to show a single block document for investigative purposes.
• Breakdown between original, edited, and unverified comments.
• Breakdown of comment payouts by creation date.

Some examples below:

The first set of testing showed issues with the active_votes process. As such I implemented a new process which adds a full verification and cross-check routine (as for the main operations block) but inserts once to Mongo only at the end of the set of active votes.

The code changes are here:
https://github.com/miniature-tiger/block.ops/commit/d55b70021b01083940913613ce3be098956898b2

Additional new features

• Additional self-validation for original posts vs edited posts.
  Self validation extended to backward loading of blocks within comment processing - i.e. any comment operation now checks for future author_payout with which to verify comment is original or edit.

• Converting string dates to timestamps
  All timestamps now recorded in MongoDB as dates rather than strings.

The code changes for the two feature above are here:
https://github.com/miniature-tiger/block.ops/commit/e5d295afdc9ee10b5c20c248bb25a820efcb20d4

• App name / version processing improvements.
  The derivation of the app name and version from the json_metadata was updated. This allows it to handle non-standard json entries (such as parley and cryptoowls).

• Post/comment split.
  The post or comment depth (0 or 1) is now captured.

The code changes for the two feature above are here:
https://github.com/miniature-tiger/block.ops/commit/3f1a7c50157657b44667d09c7b2378b371bbd524

• Improvements to the process of logging errors.
  Improvements to the process of logging errors in the blocksProcessed collection.

The code changes for this feature are here:
https://github.com/miniature-tiger/block.ops/commit/1f84d43cbac44b4965fcb922ae3e38e527939988

Progress!

---

GitHub Account

My account on github is here:
https://github.com/miniature-tiger