Block.ops - An Analysis Tool - HF20 Complications and Analysis Development
A busy week with lots of coding on two main areas for the block.ops analysis tool:
Dealing with HF20 complications
HF20 threw up some particular complications for the data collection and translation:
- The calculations of the fx rates used to convert rshares to STU vote values needed to be updated for HF20 compliance.
- The long blockchain halt on 17 September (the accidental HF20 switch) resulted in 70,000 virtual operations being attached to a single block. An entire workaround was required to allow these operations to be incorporated in the data.
Development of analyses
With the data collection and storage processes largely complete for comments, votes, and the three virtual operation reward types (author, curator, and benefactor) it was time to get the analysis functions that use this data into shape. The following analyses were created or updated:
- Vote timing analysis created: Histograms of vote timing across the life of all posts created on a particular day together with the patterns of curation reward ratios associated with the votes.
- Marketshare / Steem dApps analyses updated: Inclusion of fx rates so that combined payout results can be provided in STU rather than separate results in SBD, Steem and Vests. Numerous options were also added including separate analyses by application, choice of posts / comments, payouts allocated to dates by creation date / payout date, and aggregated results by day to allow background charts.
- Analysis of top curators by day created: Find the top curators!
- Post payout analysis for understanding payouts under HF20 and the amounts going back to the reward pool.
Bug fixes (smaller features)
- Plus some bug fixes and efficiency enhancements.
For the last year I have carried out a monthly analysis of the Steem blockchain activity by application (i.e. by the different websites and applications through which you can post to the Steem blockchain - often termed dApps).
My aim is now to build a tool that can automate such complex analyses of Steem data, providing both historic time series and rapidly updatable real-time results. In addition to the dApp analysis there are many other projects for which such a system could be useful.
This tool is Block.ops.
You can read all about block.ops (including the project aims, technology stack, roadmap and how to contribute) in the introductory post here:
https://steemit.com/utopian-io/@miniature-tiger/block-ops-an-analysis-tool-1537300276791
Repository
https://github.com/miniature-tiger/block.ops
New Features
Dealing with HF20 complications
Fx functions and improvements
The fx calculations used to convert rshares to STU vote values needed to be updated for compliance with HF20 payouts. Under HF20 rshares are passed back to the reward pool such that the total post payout and total rshares recorded in the votes are no longer equivalent. Instead the equivalence for HF20 is based around:
author payout + beneficiaries payouts = 75% rshares
Various other improvements were made to the fx operations:
- Increased robustness of derivation of fx rates through various enhancements including interpolation for gaps in steem blocks over time or lack of suitable posts from which to derive data.
- Added function to display and export prices over specific dates which allows date graphs to be produced for reasonableness checks.
The code changes are here:
https://github.com/miniature-tiger/block.ops/commit/4e714608cd8eacaf2e27673233ec084cff9a12ed
Workaround for blockchain halt
The long blockchain halt on 17 September (the accidental HF20 switch) resulted in 70,000 virtual operations being attached to a single block. An entire workaround was required to allow these operations to be incorporated in the data. The operations cannot be called through the API so have been obtained and formatted into an external json file. The process reads in the operations and pushes them through the existing functions.
Not the most elegant solution and extremely time-consuming for only one half-day of data but the only option available with this stack. Hopefully this process will only ever need to be employed for that one block!
The code changes are here:
https://github.com/miniature-tiger/block.ops/commit/4fbc2e6884752f2b76dde46c97e7d1fa1154d0f3
Development of analyses
Histograms of vote timing plus ratios of curation rewards to upvote values
A new analysis function which looks at vote timing by post duration with a particular eye on the reverse auction period and the curation rewards:
- The analysis aggregates vote values into buckets by the duration at which they are added to posts. The aggregation includes all posts in a chosen block range, for example a specific day. Buckets can be any duration, for example a histogram for each minute of the first hour or each hour of the seven day payout period.
- The overall ratio of curation rewards to upvote values within each bucket is also calculated.
- Facets are used to create analyses for various vote groups: all users, self-votes, any chosen subset of users (e.g. bidbots or top curators) entered as an array parameter.
- The analysis uses all posts (not comments since these will have much different vote patterns).
- Fx rates are applied to translate votes and curation rewards to STU for comparability.
- Results are exported to a csv file for charting.
Examples of output are shown below (figures converted to charts using mac numbers):
A full analysis (the first block.ops analysis!) can be seen here:
https://steemit.com/utopian-io/@miniature-tiger/hardfork20-curation-analysis-1539298668074
The code changes are here:
https://github.com/miniature-tiger/block.ops/commit/e447d2b977d4864ed3bd66bbec88799d0463e714
Market Share analysis update
An update with various improvements to bring this analyis function to basic production level:
- The analysis now incorporates fx rates so that combined payout results can be provided in STU rather than separate results in SBD, Steem and Vests.
- Separate analyses can now be produced by application.
- Separate analyses by date are also available to allow time series.
- Addition of options such as inclusion of posts or comments and payouts aggregated according to created date or payout date.
An example of output is shown below for the background chart of payouts across the month. The full analysis for September should be out in a couple of days. Better late than never!
The code changes are here:
https://github.com/miniature-tiger/block.ops/commit/88d067f027764e68a3c0b1f96ddd24e783045534
Analysis of top curators
The analysis of curator rewards previously found the top curation ratios (rewards vs vote values) for all votes made on a particular day or for a single user on a particular day. This has been expanded to capture the top curators over a chosen time period.
- The aggregate curation ratio stats are calculated for every user and all votes made in the chosen time period.
- Sorting these ratios gives (one possible classification) of the top curators.
- The analysis excludes "edge-cases" where curation rewards have been significantly enhanced by votes with zero weight and large rshares due to revoting. This distorts the picture with regards to vote timing.
- Fx rates lookup was included in all curation analyses to replace the previous static fx rates.
The code changes are here:
https://github.com/miniature-tiger/block.ops/commit/c652a03e8382845fe4955f83e0141e5aa59c5186
and here:
https://github.com/miniature-tiger/block.ops/commit/555bf83904b6ff4497d6c09bf6c6f2c769c68f1c
Post curation analysis for understanding HF20
An analysis which breaks down the HF20 rewards on a single post to verify the workings of the author / curator split across the reverse-auction period and illustrate the proportions of rewards being sent back to the reward pool. More to come on this one!
The code changes are here:
https://github.com/miniature-tiger/block.ops/commit/d19871479bf4fdf4a3fe694bc8fd7a3b05e097f0
Bug fixes
Small bug fix and efficiency improvements
https://github.com/miniature-tiger/block.ops/commit/ccd35a4b90d01bb8f04e9868af0460c2f252e8e2
Progress!
GitHub Account
My account on github is here:
https://github.com/miniature-tiger
Thank you for your contribution. Again a great contribution, and I really like the output of the analysis, very nicely done. Can anyone run the analysis with different parameters or is it for custome parameters.
You can also try to improve your code little bit to use .map instead of a normal for loop and also var instead of let, but other than that code looks great.
Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post, click here.
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]
Thanks @codingdefined!
All of the analyses are flexible with parameters that can be set either in the command line or in an array in the code. For example I generated the voting habits of bidbot accounts and top curator accounts using the same function, just by changing the array of accounts. The plan is to build a front-end where such lists of accounts can be selected / entered rather than having to change the array in steemdata.js. The front-end will also show which data has been loaded, list the possible analyses, and potentially have charts and tables of the results.
Thanks for the coding tips; always appreciated! JS has so many different loops it's tricky knowing which one to choose. But I haven't used .map yet - I'll have a look into that!
Thank you for your review, @codingdefined!
So far this week you've reviewed 4 contributions. Keep up the good work!
Nothing gets me hot and bothered like a good data visualisation tool, thank you for your fantastic work mt.
I've only ever run a web server (on my desktop) working on my uni php course assignment, and that was ages ago -- I have no idea about web hosting these days, so I hope you won't mind me asking:
Can I reasonably pop Block.ops on one of the free hosts? (I've found a list of things with Node.js and MongoDB support here) Or should I rather try and get it up on the desktop for a bit if I just want to ogle some graphs?
Hey @eccles, such a good question!
Part of the reason for my carrying out this project is to learn the technology. It's my first project with node.js, with MongoDB, and with the API to interact with the Steem blockchain, all of which I've got to grips with over the last month.
I also plan for it to be my first foray into servers and hosting and providing RESTful API, but as I haven't got that far yet I currently know only a little about them.
So to answer your question, I would say desktop / laptop for the moment, since that's how I'm using it, but I will be heading towards a hostable version over time.
I would also add that it's worth waiting a little until I add a front-end (currently it's all CLI run) and make it hopefully a bit more user-friendly and iron out any remaining tweaks. I also plan to add transfers, custom_json (follows/unfollows), and delegation operations fairly quickly as the most interesting of the remaining operations, so there will be more to play with. A couple of weeks and I should have it in a pretty good state for those parts.
I hope that helps. Answering a little bit from the dark!
Awesome stuff!! This is all quite a bit far out of my league, but I've been growing more and more curious about the analytics and data behind the Steem blockchain. It's all very interesting from the non-dev perspective that I have.
I just read an older post of yours from 1 year ago about visualizing curation rewards and I have a question related to that, if you don't mind.
You kind of answered it in that post, but I still feel slightly confused and maybe it's just my dull brain that's making it hard to comprehend:
Does the # of voters on a given post matter in the overall structure of payouts to the curators or is it solely based on the rShares of the votes that preceeded other votes?
In other words, if you have 100 voters on a post who each have 1000 SP or 10 voters on a post who each have 10k SP, will the curation rewards be distributed differently (i.e. the last to vote on the post getting significantly less due to the larger volume of upvotes regardless of them being the same rshares).
p.s. just wanted to say that your charts on that post made it much easier to understand! I appreciate your work greatly and I hope that with some time and a lot of caffeine I can dive into the work that you're doing with the block.ops tool!
Thanks @khaleelkazi! The Steem data is fascinating and there's so much to look at - it's just a question of being able to collect it and shape it into a form that brings it to life.
You are right. It's the total rshares (i.e. the total vote value) that matters, not the number of voters. So your curation rewards will be the same if you vote on a post that already has 10 $0.10 votes or a single $1 vote. This means you only need to look at the post payout (and the time since posting if less than 15minutes) to estimate your curation rewards.
Hope that helps!
Of course! You do a great job of bringing it to life, keep it up!
Ahh, I see. That makes it a lot simpler to me haha. I was getting super confused when I saw posts with over 1,000 voters and I was trying to calculate the curation rewards distribution and was just getting a headache lol.
It does help, thank you for taking the time to help me out!!
Thanks for the amazing analysis!
Posted using Partiko Android
No problem!
A bit off but, are you still working on Archipelago? I'd like to read about gamedev and spread the word about it.
Hey @katamori!
Yes, I'll be returning to Archipelago hopefully within the next couple of weeks. Just a little distracted currently by other projects. But soon!
Congratulations! Your post has been selected as a daily Steemit truffle! It is listed on rank 13 of all contributions awarded today. You can find the TOP DAILY TRUFFLE PICKS HERE.
I upvoted your contribution because to my mind your post is at least 20 SBD worth and should receive 144 votes. It's now up to the lovely Steemit community to make this come true.
I am
TrufflePig, an Artificial Intelligence Bot that helps minnows and content curators using Machine Learning. If you are curious how I select content, you can find an explanation here!Have a nice day and sincerely yours,
TrufflePigHi @miniature-tiger!
Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your post is eligible for our upvote, thanks to our collaboration with @utopian-io!
Feel free to join our @steem-ua Discord server