What is the reward pool share of vote sellers, bid bots and autovotes?

in utopian-io •  3 months ago

Repository

https://github.com/steemit/steem

Introduction

We've had a couple of posts already about the reward pool share of the bid bots and paid voting services. Previous analyses gave a reward pool share varying between 25 to 30% of the total daily vote values. The approach is rather straight forward: query all votes from the blockchain or a database, build a list of all voting bots and check which fraction of votes was done by them. What's not included in this number is the number and value of votes sold via sell-your-vote services like @smartmarket and @minnowbooster. I'm not aware of published numbers on the number and volume of votes with services like these.

Getting the numbers for sell-your-vote services is hard, because regular user accounts are used to vote. The vote seller receives a share of the bid in return. There may be some accounts that basically sell all their votes that way, but a lot of them use this as a side incoming when their VP would be idle. These accounts also do manual/other votes, a simple account list doesn't work there, not even talking about getting this list in the first place. Looking into transfer history was suggested a couple of times already. Whoever tried that knows that this is a big mess to dig through. Invalid bids, refunds, and most importantly: user balances managed on smartsteem.com and minnowbooster.com. There's not necessarily a blockchain transaction, neither for the bid, nor for the vote seller payment.

However, there's a way: The signatures of blockchain vote transactions can reveal who initiated the vote on behalf of the users. As a positive side effect: the same method also reveals autovotes via steemauto or steemdunk. The downside is: this approach is much more compute intensive and therefore slower than the bot list approach. Analyzing larger time ranges is costly.

Outline

  • Scope of the analysis and tools
  • Transaction signature 101
  • Most frequent vote signers
  • Share of sell-your-vote, bid bots and autovotes by value

Scope of the analysis and tools

All data was queried directly from the blockchain using batched appbase calls with beem and saved in a local MongoDB database for further processing. All query and processing scripts are done with python.
The analysis covers all votes done on Aug. 23rd 2018, one full day. The reason for the small time range is the amount of post processing required to get the results.
All scripts to generate these results are in my GitHub.

Transaction signature 101

Every interaction with the blockchain is cryptographically signed. Private keys are used to sign all your votes, comments, transfers, etc.. If you click the upvote button on steemit.com, your browser assembles a data block containing (among others) your account name, the post you want to vote for as well as the upvote percentage. This data is serialized and signed with your private posting key. The resulting signature is appended to the data block and sent to a Steem node. The Steem node can now verify that this transaction is actually signed with your private key based on your public key. It obviously only includes transactions into the blockchain where these signatures are valid and the signer is allowed to sign that kind of transactions.

However, there's more: authorized accounts. If you authorize an account to post in your name, that account is allowed to sign transactions in your name, with it's own private key. This means, based on the signature appended to each blockchain transaction we can check which key was used to sign a transaction. The public key of the signer can be extracted from the signature.

Coming back to the original topic: If you sign up with smartsteem or minnowbooster to sell your vote, you authorize @smartsteem or @minnowbooster to post/vote in your name - with their key!
If you sign up with SteemAuto or SteemDunk, you authorize @steemauto or @steemdunk to post/vote in your name - with their key!
By extracting the public keys of the signers from the signatures on the vote operations, we can determine if the vote was signed from the account's own private key, or any authorized app.

In order to get this information, full transactions are required. The signatures and the transaction raw data is not available from SteemSQL. This has to be retrieved directly from the blockchain. Batched appbase calls help to speed this up, but it's still magnitudes slower than querying SteemSQL.
Once the transactions are available, the public key of the signer can be extracted with beem's Signed_Transaction class. This takes a couple of 10ms per transaction on my machine. There are ~700k votes per day, so that's going into hours of computing time per day even in the optimal case. I failed to speed it up with parallel threads or processes (GIL or my incompetence?), but there may be room for improvement...
Now we have the signer's public key, the voter, the voted post and the voting percentage - what's missing is the vote value. In order to get that, we need a get_content lookup for each and every post and comment being voted on that day (around 60k). Luckily, there are again batched appbase calls available to get 1000 posts at once per call, but it still takes time...

Most frequent vote signers

These are the top 10 vote signers on the given day from a total of 695k votes from 51k distinct voters:

Rank# VotesPublicKeyAccount
1286383STM8W...['steemauto']
246210STM7y...['smartsteem']
342362STM5k...['steemconnect']
410410STM5T...['steemdunk']
54433STM6n...['upvotebank']
..........
91526STM6H...['minnowbooster']

This list is led by steemauto with 268k votes! This means that more than 40% of all votes (by number) on that day were done by steemauto. That's quite a lot, but I'm actually not really surprised. I would have expected even a higher number there, but this is not the only auto-vote service. Smartsteem is second with around 46k votes or 6.5% (by number). The third entry, @steemconnect, contains the votes done by users via several of the dDapps, including busy.org, dtube, dlive, and others. It may also include auto-votes. So these are at least partly "manual", partly auto-votes. Unfortunately, the same key is used for all these dApps, so we can't distinguish votes from different dApps there. @steemdunk is another auto-voter with around 10k votes. @upvotebank is a vote-for-a-vote service. @minnowbooster is ranked 9th by the number of votes here.

A couple of other auto-voters like @steemvoter cannot be shown here, because they use the posting key directly. These votes cannot be distinguished from manual votes on steemit.com by the signature.

The votes from @smartsteem and @minnowbooster in the table above also contain those votes done directly with the main accounts, so these numbers are not only vote selling operations.

Share of sell-your-vote, bid bots and autovotes by value

The number of votes is only one aspect, more relevant is the value of the votes. The following table includes the relative share of the vote signers by vote value, namely the sum of rshares in relation to all vote rshares of that day:

SignerRelative Vote countRelative vote value
Bid-bots/paid bots1.03%28.17%
Smartsteem vote sellers6.58%3.82%
Minnowbooster vote sellers0.21%1.44%
Steemauto auto-votes41.17%6.81%
Steemdunk auto-votes1.50%0.26%
Steemconnect (dApps)6.09%4.21%
Upvotebank (vote-for-vote)0.64%0.02%

This tables shows the share of bid-bots and paid voting services in the first line. This share was determined from a list of known bots. If a vote from @smartsteem is signed by @smartsteem, this is contained in the "Bots" line. If a vote from a vote seller is signed by @smartsteem, this vote is contained in the "SmartSteem" line.

The overall share by value of the paid voting services for that given day is at 28.2%. Past analysis have shown values between 25 and 30%, so this didn't change very much since then. @smartsteem's sell-your-vote operations make up 3.8% of all votes by value and @minnowbooster adds another 1.4%. This means, that another around 5% of all votes by value on top of the 25-30% of the bot votes are sold votes. There's an interesting difference between MinnowBooster and SmartSteem in the ration between the relative share by count and value: 6.5% of votes by SmartSteem make up 3.8% of the value, while 0.2% of the MinnowBooster votes make up 1.4% of the value. This means that the MinnowBooster vote sellers in average make larger votes than the SmartSteem vote sellers.

The value of the auto-voters on the other hand are small compared to their relative number. SteemAuto makes more than 41% of all votes by number, but those generate "only" 6.8% of the total vote value. The auto-votes are in average much smaller than the sold votes. This not very surprising, given the SP distribution across active accounts.

Conclusions

  • The votes from bid-bots and paid voting services sum up to 28.17% of all votes by value for that day. This is consistent with my previous analysis giving values between 25 and 30%, however with fluctuations across days.
  • Sell-your-vote operations currently make up around 5% of the total vote value per day, on top of the 25-30% done by bid-bots and paid voting services.
  • Two of the auto-votes together make up more than 42% of all votes on that day by number. With several other auto-vote services using the posting key directly, or contained in the SteemConnect line, the real number is very likely to be much higher.

In general, the data set is comparably small due to the complex post processing. There's certainly room for improvement on the processing speed, which would immediately improve the value of the results. This is the first estimation on the vote selling business that I'm aware of. If there are others, please leave a comment.

I had absolutely no idea which order of magnitude to expect for the vote selling part when starting this analysis. Is around 5% much?


Edit:
as pointed out by @therealwolf in the comments below, Smartsteem also signs a "solid number" of votes from vote sellers with the Steemconnect key. This means that an unknown fraction of the 4.21% of votes by value, currently assigned to Steemconnect, are actually also paid votes from Smartsteem vote sellers. This makes vote selling a share of something between around 5-9% of all votes by value on that day, on top of the 28.17% directly done from bid-bots. Thanks for this remark!


GitHub Account

https://github.com/crokkon

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  
Loading...

Bid-bots/paid bots 1.03% 28.17%

If I’m not mistaken, concentration of stake/influence was one of the main gripes around here prior to last year’s hard forks that were supposed to help correct it. But it appears that the changed/implemented protocols have failed to make much of a difference at all in that regard.

Two of the auto-votes together make up more than 42% of all votes on that day by number.

I think it’s safe to say that adding Steemvoter and other private automated voting would put this figure well over 50%, possibly into the 60s or more.

So, what we have here is a platform with activity that is dominated by automation and largely influenced by paying for what are essentially whale votes that are actually larger than most non-bidding (human) whales. To me, this is much worse than any superlinearity “problems” that previously existed. I’m not sure how a social media platform can survive like this if it continues down the same path.

But since very few seem to care - particularly the “leadership” that created the original problems, then created the worse “solutions” to the original problems they created - I don’t believe any of this data will matter much. We all know that something isn’t right, but instead of seriously looking at this data and trying to comprehend it, most people here will simply cheerlead and shill for Steem and STINC, then denigrate anyone who isn’t doing the same.

And the problems continue...

·

Thanks for stopping by, @ats-david. I can't comment on pre HF18/19 since I wasn't there at that time, but you pretty much summarized the state of Steem as it is now. The platform is definitely largely automated, and my numbers here only address two aspects of many...

·

You are so correct my friend.
I want to invite you to participate in NFL Pick'em this year.
Thanks for considering...
https://steemit.com/sports/@steemitnflpickem/steemit-nfl-pick-em-week-one-win-steem

After all that time it was right there using keys. Respect. This is an amazing analysis. 5% is not too bad. But the 41% autovotes hmmmm thats why no one is reading. Very interesting

Need to learn to use beem. Holger pointed me to it before, just need to find the time.

Awesome work.

Posted using Partiko Android

·

That is actually concerning. The contents have moved beyond from just reading and people aren't consuming.

Taking "don't read half the times" to a literal level.

Oooh great! This is very clever! I was doing this locally earlier by tracking memos, though it doesn't handle when people are using internal balances to initiate a vote (I do handle refunded balances though). This, while time consuming, should capture everything. Thanks for the analysis!

Posted using Partiko Android

·

Thanks, @eonwarped! Yes, the internal balances make the result from the memo approach only a lower boundary. Another limitation with memos is that you typically know how much was spent for the votes, but can only estimate the vote value based on typical ROIs. Nevertheless, it scales better to larger time ranges.
Would be interesting to know how many bids are done from internal balances vs. blockchain transfers! :)

Hey, @crokkon!

Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!

Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).

Want to chat? Join us on Discord https://discord.gg/h52nFrV.

Vote for Utopian Witness!

Very interesting indeed, takes some time to assimilate all these numbers and figures... Well done thanks for posting!
/FF

In your conclusion you wrote:
The votes from bid-bots and paid voting services sum up to 28.7% of all votes by value for that day

But doesn't it sum up to 28.17+3.82+1.44=33.43%?

In general this is really a great investigation as it solves a long time outstanding question!

·

Thanks for your remark and cross-checking the numbers! A '1' got lost there: 28.7 -> 28.17.
The 28.17% are all votes directly from the bid-bots and paid voting services. On top of that are 3.82% + 1.44% from the vote sellers via smartsteem and minnowbooster.
In total 33.43% for bots + vote sellers for that specific day.

Interesting stuff, @crokkon. Very, very interesting!

I want to mention that Smartsteem previously had the steemconnect option and so far, still a solid number of votes are being made through steemconnect instead of posting key signing.

Which means, the number for Smartsteem will be a bit higher. And it would be quite interesting to have data for multiple days or even weeks.

Also, while I don't have an exact number for daily votes, so far over 6.6 million votes have been made from vote-sellers.

Anyway, thank you for your work!

·

Thanks, @therealwolf, also for the additional information! So this means that at least a part of the 4.2% of votes (by value) currently assigned to steemconnect actually are smartsteem vote sellers as well. I agree that it would be interesting to see the picture on a larger time scale.

Great analysis, @crokkon ! Congrats!

·

thx @verodato, also for the RS!

Automation isn't that bad if used properly.
I'm still doing full manual curation but I'm well aware of how inefficient is that. However, there's a chance here for some middle ground: https://steemit.com/wise/@lukestokes/sneak-peak-wise-vote-an-interview-with-noisy

·

Kudos for the full-manual curation, great to see that this works out also with higher stakes!
Wise is a very interesting project for finer-grained voting controls. I haven't heard much about this project lately, but it's certainly worth a more detailed look.

Great analysis. No wonder it's taken you so long to post since last time!

What I take away from those two pie charts are the small no.of votes from bid bots compared to large overall vote value in relation to the large number of steem auto votes compared to relatively small value.

So this must be case of a few wealthy players buying themselves votes via bid-bots and then the medium to small level players spreading their votes relatively thin via auto voters.

I don't think personalized auto voting is all bad, as long as you review it every few weeks!

Also aren't smartsteem and minnowbooster basically 'bid-bots' too, essentially?

·

Thanks, @revisesociology! I was offline for a couple of days, so it wasn't all about preparing this post ;)
Your take-away is correct. I agree, that auto-voting isn't bad per-se. I use it as well and I don't see a general problem with it. I think we all knew before that a good fraction of votes is automated, seeing some numbers gives a bit more of a feeling on the orders of magnitude.

Also aren't smartsteem and minnowbooster basically 'bid-bots' too, essentially?

I'm not sure I fully understand. Direct votes from these accounts are included in the bid-bot/paid votes column. What was unknown before is the number and value of votes that were cast via private accounts using those two vote-selling proxies.

·
·

Oh OK - I must have just misread... about the SS and MB being a subset of bots....

Still, it's not actually as botified as I thought.

Cheers!

Posted using Partiko Android

If bidbots take 30%, and the top ten voters take 30%, how much is left once the top 100 voters vote?

·

and the top ten voters take 30%

Interesting aspect! Is this a number you've seen elsewhere calculated, or your assumption / guess?

Congratulations! Your post has been selected as a daily Steemit truffle! It is listed on rank 7 of all contributions awarded today. You can find the TOP DAILY TRUFFLE PICKS HERE.

I upvoted your contribution because to my mind your post is at least 15 SBD worth and should receive 196 votes. It's now up to the lovely Steemit community to make this come true.

I am TrufflePig, an Artificial Intelligence Bot that helps minnows and content curators using Machine Learning. If you are curious how I select content, you can find an explanation here!

Have a nice day and sincerely yours,
trufflepig
TrufflePig

Hi @crokkon! We are @steem-ua, a new Steem dApp, using UserAuthority for algorithmic post curation! Your post is eligible for our upvote, thanks to our collaboration with @utopian-io! Thanks for your contribution, keep up the good work, and feel free to join our @steem-ua Discord server

Great post, I'm glad I dropped by to give you this invite:
Football season is here, join NFL pick'em.
https://steemit.com/sports/@steemitnflpickem/steemit-nfl-pick-em-week-one-win-steem