Block.ops - An Analysis Tool - All things votes, robustness, and self-validation.

The good news with HF20 was it made lots of time for coding!

This contribution covers all things voting, including:

• Processing of vote and curator reward operations as well as grabbing extra information from active votes, and
• Adding the functionality for analyses of curator rewards.

In addition I looked to increase the robustness of the tool by:

• Adding robustness to operations processing, and
• Working on re-processing and solving the edited comments issue.

---

For the last year I have carried out a monthly analysis of the Steem blockchain activity by application (i.e. by the different websites and applications through which you can post to the Steem blockchain - often termed dApps).

My aim is now to build a tool that can automate such complex analyses of Steem data, providing both historic time series and rapidly updatable real-time results. In addition to the dApp analysis there are many other projects for which such a system could be useful.

This tool is Block.ops.

You can read all about block.ops (including the project aims, technology stack, roadmap and how to contribute) in the introductory post here:
https://steemit.com/utopian-io/@miniature-tiger/block-ops-an-analysis-tool-1537300276791

---

Repository

https://github.com/miniature-tiger/block.ops

---

New Features

Votes, active votes and curator rewards

Blockchain data on votes can be found in (at least) three places. The block transaction "vote" which is included in a block when the vote is made, the virtual operation "curation_reward" which is attached to blocks at post payout, and the aggregated vote data for a post which is updated throughout a post's life and can be extracted using the "get_active_votes" API call.

This new feature adds (or, in the case of "curation_reward", reworks) the functions to extract this data from the blockchain and add it to the MongoDB. The vote data is stored in each MongoDB comment document as an array of objects, with the fields of the objects drawn from the three different sources. The "get_active_votes" call is triggered by the post payout (past this point any updates are irrelevant).

The data structure of the votes section for a single comment within MongoDB now looks like this:

Different parts of each curator object are drawn from the three separate sources.

The code changes are here:
https://github.com/miniature-tiger/block.ops/pull/3/commits/216da6338231dbed9e3c602fc37a6eae680201ce

Adding analysis of curator rewards

Adding the vote information to each post/comment allows numerous new analyses to be carried out. The first vote analysis I have implemented looks at the curation rewards earned for each vote made using the ratio of curator_reward vests to voter rshares (i.e. curation reward / vote size).

The analysis calculates the ratio for all curator rewards in a chosen date range and then sorts them in order. The top 10 are output in the console.log, although all result are available. The analysis can be carried out across all accounts or for a single specified account (entered as the third parameter in the command line).

So here are the top ranking "curation rewards vs vote size" results for 24 September (just before HF20) for votes above 50bn rshares.

And here are my own top 5 from the same day - all Utopian posts!

The SP_reward and STU_vote_value amounts are approximate, based on hard-coded currency fx until I implement the Steem/SBD/STU/rshares/vests fx tables.

The code changes are here:
https://github.com/miniature-tiger/block.ops/pull/3/commits/27ccb861c4073438f6f73c0dfb4a98c45a5a6e5d

Adding robustness to operations processing

Previously the tool logged each block processed in a MongoDB collection (which I creatively entitled the "blocksProcessed" collection). To improve robustness the logging has been updated to collect the status of each individual operation processed rather than each block processed (a single block includes many operations). This will allow greater robustness in ensuring the data collected from each block is complete.

An example of the revised log is included below (this block is still missing one operation so its status is 'processing' rather than 'OK'.

The code changes are here:
https://github.com/miniature-tiger/block.ops/pull/3/commits/241cd74b89bb395706a5ef2c2bc64179d3b5d517

Working on re-processing and solving the edited comments issue

One of the issues arising with my previous marketshare analyses has been that the json_metadata is changed on posts (and comments) when it is edited. This problem should no longer arise since I am now obtaining the original json_metadata from the block operation when the post is created.

However a similar problem arises in that there are difficulties in differentiating original comment operations (where comments includes both posts and comments) and comment edit operations. Original 'comment' operations and operations which edit those comments appear the same in blocks, with no clear marker to distinguish them.

This issue is exacerbated by the tool being applied to blocks in any order - a necessity for an analysis tool.

Options to get around this issue are:
(1) Pull the comment itself using get_content and check the created date. This results in pulling much of the comment data twice (in the block operations API call and in the get_comment call) and potentially could result in data limits from the node being exceeded. Not an option.
(2) Always run the tool forwards starting from inception or from a point in time 7 days before the analysis period commences. Not great.
(3) Self-validation. The data, if run over a long enough period or if enough data is collected, is largely self-validating. Author_payout operations can be used to validate whether a comment is the original or an edit, based on whether the time between the comment and author payout timestamps is 7 days.

This feature implements this validation (or at least makes a first version of it) as well as various other re-processing robustness enhancements.

The code changes are here:
https://github.com/miniature-tiger/block.ops/pull/3/commits/c5d01ce7139fb32557521c803743d5c163e6889a

Progress!

---

GitHub Account

My account on github is here:
https://github.com/miniature-tiger