Block.ops - An Analysis Tool - Steem Currencies and Testing
Some hectic days of coding, trying to get this project up and running for some month end analyses. I'm running a little late!
This contribution mainly covers:
Implementation of Steem currencies and their exhange rates:
The blockchain data is expressed in many "currencies" (Steem, SP, Vests, SBD, STU, rshares) but most analyses will want results in a single currency, typically STU (what we see on post payout and vote amounts) or Steem. This new feature provides exhange rates between the various currencies.Testing / validation to ensure data loaded is complete and robust.
Testing of my first batches of loaded data revealed a number of issues. This feature adds a number of tests and validations to ensure the loaded data is complete and robust. I also enhanced the active_votes logging process as this was the main cause of the issues.
In addition I added a number of other new features (mainly smaller features):
- App name / version processing improvements.
- Post/comment split.
- Improvements to the process of logging errors.
- Converting string dates to timestamps
- Additional self-validation for original posts vs edited posts.
For the last year I have carried out a monthly analysis of the Steem blockchain activity by application (i.e. by the different websites and applications through which you can post to the Steem blockchain - often termed dApps).
My aim is now to build a tool that can automate such complex analyses of Steem data, providing both historic time series and rapidly updatable real-time results. In addition to the dApp analysis there are many other projects for which such a system could be useful.
This tool is Block.ops.
You can read all about block.ops (including the project aims, technology stack, roadmap and how to contribute) in the introductory post here:
https://steemit.com/utopian-io/@miniature-tiger/block-ops-an-analysis-tool-1537300276791
Repository
https://github.com/miniature-tiger/block.ops
New Features
Implementation of Steem currencies and their exhange rates
The "currencies" of the Steem economy are complex!
The blockchain data is expressed in many currencies. Payouts for authors, curators and benefactors are recorded in a mixture of SBD, Steem and Vests (for Steem Power). The vote data uses rshares as the metric for vote size. The comment data and user interfaces on dApps use STU (i.e. SBD at $1 rather than market price) to illustrate vote sizes and post payouts.
Most analyses will want the results in one currency. Typically this would be STU since this is the measure most commonly displayed by the Steem dApps. However the blockchain API give no access to historic fx rates between these different measures.
This new feature solves this problem, providing exchange rates between the various currencies, allowing the results of analyses to be presented in a user-friendly format.
There are a number of options that could have been used to achieve this:
- Use the data to implement the full workings of the reward pool, the 3.5 day median witness feed averaging, and the steem price inflation. A project for another day!
- Extract the data from every post and comment in addition to every block operation so that the payouts are available in all currencies. Pretty inefficient for historic analyses.
Instead I have implemented an "empirical" solution. This chooses a single suitable post for each hour (according to various criteria), pulls the data for that single post and uses the payout data to derive fx rates. For example the curator payout in STU from the post can be compared to the total vests from the curator_reward payouts from the virtual block operations to give a Vests/STU fx rate for each hour. A similar approach is applied for Steem and rshares fx rates.
This approach allows a table of hourly fx rates to be built which can then be applied to all payouts.
These charts show the results of this process for data between 1 September and 11 September 2018. All rates are against 1STU (so may show the inverse of what you are used to seeing).
There are a couple of blips on the charts which require investigation but otherwise the results looks sufficiently stable for use.
The code changes are here:
https://github.com/miniature-tiger/block.ops/commit/4eb3eeb2b5fcb0de51b18a4691b8dc87841d9a84
Testing / validation to ensure data loaded is complete and robust
Testing of my first batches of loaded data revealed a number of issues. This feature adds a number of tests and validations to ensure the loaded data is complete and robust.
Numerous checks were implemeneted or upgraded to validate the cohesiveness of the data. Including:
- Report Blocks Processed: Upgraded to compare the blocks processed on each day against the total blocks that exist for that day, as well as the breakdown between ok, processing, and error statuses. As an example:
- Block Check Total: Check total on each block to validate all operations have been processed and uploaded (with allowance for skipped operations).
- Check Total of active vote processing: Active votes processed and logged vs active votes on the comment.
- New function to show a single block document for investigative purposes.
- Breakdown between original, edited, and unverified comments.
- Breakdown of comment payouts by creation date.
Some examples below:
The first set of testing showed issues with the active_votes process. As such I implemented a new process which adds a full verification and cross-check routine (as for the main operations block) but inserts once to Mongo only at the end of the set of active votes.
The code changes are here:
https://github.com/miniature-tiger/block.ops/commit/d55b70021b01083940913613ce3be098956898b2
Additional new features
Additional self-validation for original posts vs edited posts.
Self validation extended to backward loading of blocks within comment processing - i.e. any comment operation now checks for future author_payout with which to verify comment is original or edit.Converting string dates to timestamps
All timestamps now recorded in MongoDB as dates rather than strings.
The code changes for the two feature above are here:
https://github.com/miniature-tiger/block.ops/commit/e5d295afdc9ee10b5c20c248bb25a820efcb20d4
App name / version processing improvements.
The derivation of the app name and version from the json_metadata was updated. This allows it to handle non-standard json entries (such as parley and cryptoowls).Post/comment split.
The post or comment depth (0 or 1) is now captured.
The code changes for the two feature above are here:
https://github.com/miniature-tiger/block.ops/commit/3f1a7c50157657b44667d09c7b2378b371bbd524
- Improvements to the process of logging errors.
Improvements to the process of logging errors in the blocksProcessed collection.
The code changes for this feature are here:
https://github.com/miniature-tiger/block.ops/commit/1f84d43cbac44b4965fcb922ae3e38e527939988
Progress!
GitHub Account
My account on github is here:
https://github.com/miniature-tiger
Thank you for your contribution. Again a great contribution with details about all the code you have done. I really liked how you write about your commit messages, very descriptive.
Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post, click here.
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]
Thank you for your review, @codingdefined!
So far this week you've reviewed 13 contributions. Keep up the good work!
Congratulations! Your post has been selected as a daily Steemit truffle! It is listed on rank 10 of all contributions awarded today. You can find the TOP DAILY TRUFFLE PICKS HERE.
I upvoted your contribution because to my mind your post is at least 11 SBD worth and should receive 123 votes. It's now up to the lovely Steemit community to make this come true.
I am
TrufflePig
, an Artificial Intelligence Bot that helps minnows and content curators using Machine Learning. If you are curious how I select content, you can find an explanation here!Have a nice day and sincerely yours,
![trufflepig](https://steemitimages.com/640x0/https://raw.githubusercontent.com/SmokinCaterpillar/TrufflePig/master/img/trufflepig17_small.png)
TrufflePig
Hi @miniature-tiger!
Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your post is eligible for our upvote, thanks to our collaboration with @utopian-io!
Feel free to join our @steem-ua Discord server
Congratulations @miniature-tiger! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :
Click on the badge to view your Board of Honor.
If you no longer want to receive notifications, reply to this comment with the word
STOP
Do not miss the last post from @steemitboard:
Hey, @miniature-tiger!
Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!
Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!