Clarifying Roles And Expectations In The STEEM Ecosystem - A HF 20 Post-Mortem

in witness-category •  18 days ago

From the Wikipedia definition:

A project post-mortem is a process, usually performed at the conclusion of a project, to determine and analyze elements of the project that were successful or unsuccessful.

As HF 20 settles slowly into place, I decided to write a short piece about the current status of the ecosystem. There is a lot of confusion around the roles of the people involved, and most humans, unfortunately, are wired to react with frustration to any confusing situation, instead of careful analysis.

Who Runs The STEEM Blockchain?

My answer to this question is: witnesses. What is a witness? According to the white paper:

Conceptually, the consensus algorithm adopted by Steem is similar to the consensus algorithm adopted by companies throughout the world. People with a vested interest in the future value of Steem vote to select individuals responsible for including testimony in the public record. Voting is weighted proportional to each individual's vested interest.

So a witness is an individal responsible for including testimony in the public record.

Who Builds And Maintains The Software For The STEEM Blockchain?

The software powering the STEM blockchain is open source. So, theoretically, everybody can build and maintain the software for the STEEM blockchain (there are already some forks of the main repository, both public and private).

But, in reality, the software is built and maintained by Steem INC, a company which not only launched the initial product, but it's also highly invested in the ecosystem. Accounts owned by Steem INC, like the @steemit account, are still responsible for a large part of the functionality of the ecosystem. Without @steemit paying for new accounts there will be no growth (the new account creation mechanism changes this in the right direction, allowing other high stake participants to create accounts on their own).

So, we have two main roles:

  • builders of the software
  • witnesses (enforcers of the software)

What Is Required For A STEEM Witness?

Here are just a few qualities (or characteristics, if you want) required for a witness to fulfill his role:

  • 24/7 hardware availability
  • moderate software knowledge (to build, install and maintain the nodes)
  • political skills (to create and maintain alliances with relevant stakeholders - for securing his position from whales)
  • communication and community building skills (to secure his position from dolphins, minnows and redfish)

Some witnesses have them all, while some of them have only a part of them and rely on association or delegation to fulfill the rest.

What Is Required For A STEEM Developer?

  • strong C/C++ coding skills
  • strong cryptography skills
  • strong knowledge of distributed systems
  • strong testing skills in distributed systems
  • loyalty to the employer

In my opinion, all of these skills must be in top 1% of their respective category to ensure a consistent and high quality product. If any of these is lower than that, then we may get into trouble.

Is There Any Overlapping?

In an ideal world, the two categories will be completely separated, because any category will be sufficiently incentivized to maintain its role.

But then again, in reality, there is significant overlapping. Some of the developers are also witnesses and some witnesses contribute to the development from outside the company (open source model makes this easier).

This overlapping may be economically viable for some of the participants, but it creates significant liabilities. For instance, if a dev is also witness, he can strike the balance in favor of a new hardfork.

Something like this happened in HF 20: the software wasn't sufficiently tested, but, since in the first 20 witnesses there are a few devs, and since the rest of the witnesses blindly trusted the devs, the public version was released at a lower quality than required. So low that the blockchain first halted (an extremely serious event) and then was rendered unusable by a bug in the RC plugin which didn't let people actually interact with the blockchain.

Is The Role Overlapping The Only Problem?

Again, just in my humble opinion, I don't think so. Role overlapping increases the confusion and can lead to so called "black swan events", but it's not the only problem.

The other, most important part of the deal is "expectation".

What's expected from a witness? To just be responsible for including testimony in the public record?

What's expected from a builder? To just build, test and ship code?

What's expected from Steem INC? To keep paying for new accounts? To steer the development in the direction they consider relevant? To hire more devs?

And then, when some of these expectations are not met, what's happening? Are there any real life consequences for this?

All in all, STEEM is just a tokenized social media business, so, in the grand scheme of things, if everybody does their job well, and the product is good, the value of the token should rise. And if things aren't going well, the value of the token should go down.

But it's this the only incentive that we have? It's this the only way to measure the quality of our efforts and involvement?

I don't think so.

I think we need better processes, and these processes should be built on readjustment of both roles and expectations.

So, Which Way To Go From Here?

HF 20 was a failure, one that it is slowly solved by small, incremental patches. But a failure.

The main cause of it was, in my opinion, the lack of a proper testnet, one that could be accessed and investigated by witnesses, in their capacity of the first beneficiaries of any hardfork. Two years after launch, to run a blockchain based social media platform without a testnet was just inconceivable, a time bomb. And, with HF 20, time ran out and the bomb exploded in everybody's face.

Let's stop for a second and think about that. It's not that if something goes wrong only devs are to blame, or only Steem INC, or only witnesses. We all suffer. Just like when things are good, we're all getting the benefits.

We're all in this together, wether we like it or not. So we have to solve this shit together.

Here are my proposals for the next stage of this (let's face it, very interesting so far) experiment:

  • Consensus changes (enforced by code)
    -- increase the amount of core witnesses from 20 to 40 and reassign the rewards accordingly (if now a top 20 witness gets around 200 STEEM / day, decrease it to 100 STEEM / day) - reason: the top 20 is becoming inertial and easy to control, also the lower the number of people who control the direction, the bigger the attack surface by bribe, negligence, etc. A bigger community of core witnesses will be more balanced. Also, from a technical point of view, that will give 2 minutes to restart your node if you have troubles, instead of 1.

  • Non-consensus changes (non-enforceable by code)
    -- dedicated testnets for various versions of the blockchain. reason: we need to test stuff before we release it. In case you didn't get it from the first time: we need to test stuff before we release it. And yet again, for those with a short memory: we need to test stuff before we release it.
    -- transparent communication places for witnesses and devs (slack, chat, whatever) enforced by turn based moderators to avoid deviations and FUD. reason: communication places now are more or less casual and it's easy to stir up frustration and unproductive talking. The need for communication between devs and witnesses is huge.
    -- pre-approved features for each hardfork, use a tool for voting all the features that we want to go in the next hardfork (as opposed to just write a post wether or not you agree with it). reason: too many features are increasing exponentially the risk for bugs, not enough features may jeopardize progress


I'm a serial entrepreneur, blogger and ultrarunner. You can find me mainly on my blog at Dragos Roua where I write about productivity, business, relationships and running. Here on Steemit you may stay updated by following me @dragosroua.

Dragos Roua

Wanna know when you're getting paid?

I know the feeling. That's why I created, an easy to use and accurate tool for calculating your Steemit rewards

It's free to use, but if you think this is a useful addition, I'd appreciate your witness vote.

Thank you!

Psst: new to Steemit? Start Here

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

My thought is that Witness role is a really important work, more than devs role. They are the judges of the system and they have to behave as so. Looks to me that is very easy for the 20 first do nothing and keep their power here... because, seems that they defend each other...
A better and more balanced witness reward within the 50 first witness would help in order to increase democracy...
About the test... I don’t want to add anything...this failure cannot happen again

In a couple of months we will consider this period as period of growing pains.


I so wish you were right :)

My take away: we need to test stuff before we release it ! :-)

Very good post ! The problems you spot are real and important. They might not be the only problems, nor even the most important though. The "consensus change" you propose is motivated by good intentions but doesn't sound realistic - you'd need to have the top 20 voting in favour of ... decreasing their own power and rewards. Ever met turkeys who have voted for Christmas? The technical benefit can be obtained by increasing block-time to 6 seconds and doubling blocksize (while doubling the top 20 reward).

Non-consensus changes: who pays for the testnets? Transparent communication loosens the tight grip Stinc holds over the Top 20. Stinc can play the political games of the Communist Party and use its very significant steem power to "beat into submission" and turn the Top 20 into puppets (from the HF20 experience, it is the current situation)

They could indeed become more professional and go for a tool that helps into managing the set of features going into the next HF ...


The technical benefit can be obtained by increasing block-time to 6 seconds and doubling blocksize (while doubling the top 20 reward).

Yep but user experience would get worse... Better we decrease our rewards , because in reality indirectly we increase it. I want steem @ $10 with "less" rewards instead @ $1 with "huge" rewards :)


THAT is the spirit! A small part of HUGE PIE is still more than a BIG part of a tiny little pie


EXACTLY THIS ^^^ couldn't say it better !


Ever met turkeys who have voted for Christmas?

haha well said!

i'm definitely voting for this:

Transparent communication loosens the tight grip Stinc holds over the Top 20.

how to get it to come about though? stinc and t20 are notoriously poor at communication...

I know, most users do not understand most of the tech issues, I include myself into that group. But maybe, at some point, some of this tech improvements could be voted from the users. Even if it is just from a pure "opinion" standpoint, and even if a very small portion of the users vote, it would be a nice piece of information to consider when deciding priorities to development areas.

Great clarification, you are doing great work on your site i use it almost every day, keep up

thank you for clearing some more things up for me! i am glad i vote for you- always appreciate your straighforward manner and clear communication (something we don't get from a lot of witnesses.)

increase the amount of core witnesses from 20 to 40 and reassign the rewards accordingly (if now a top 20 witness gets around 200 STEEM / day, decrease it to 100 STEEM / day) - reason: the top 20 is becoming inertial and easy to control, also the lower the number of people who control the direction, the bigger the attack surface by bribe, negligence, etc. A bigger community of core witnesses will be more balanced.

as an active user, i 100% vote for this.

inertial is exactly the thing that is wrong with the t20 pool- that is never good for a business or any type of organization!

what can we do to further the implementation of this idea?


Thank you. For now, I think disseminating the idea to get the opinion of as many users as possible is the way forward.

Which witnesses are also devs? I want to take my vote from them and I hope many others will do the same.


I don't think it's a good and constructive way of acting, so I'm not going to tell which are which. I don't think putting blame on people is the way forward and some devs / witnesses had (and still have) significant contributions to the ecosystem. I certainly don't want even more polarization between any factions around here. I described the roles in order to get a better understanding of how Steemit, as a whole, works and not to start some mob-style cleanup. Hope you understand.

Thanks @dragosroua. You have the ability to explain even for the techy stupid people all that stuff what lies under steemit's shining surface.
I assumed that you are already a witness, but not. So I voted for it. Who if not you could fit into that position. Good luck!

Damn. I like the way you think. No wonder I vote for you.

I think the increase in consensus witnesses just might work. I like the idea. I've spent a fair amount of time trying to figure out the right voting protocol but this might be a way better idea. Still thinking.

I have a couple of questions I have started asking, so you get them too.

What the hell is up with the numbers of Manna and RC? TRILLIONS? A number best comprehended using scientific notation?

What it looks like to me is a 'baffle them with bullshit' number. Any value can be quantified into a 4 digit number that ordinary human brains can comprehend. Trillions? Sounds like the US Government. They use the really big numbers to dumb down the electorate.

Did you ever consider that maybe the whole HF20 deal was really to start a Star Trek style calendar? Todays date becomes HF20.05 in that scenario.


Big numbers in MANA allows for a more granular quantification. Steem Power works the same way, it's based on vshares, which are also denominated in huge values and then translated in a more human-readable format.


I get that. If I want to land a probe on the moon I want to know how many meters it is precisely on the day and time of landing. If I'm looking through my telescope 239 k miles is good enough. Every single day. 12 or 14 digits is more information than I need and caused horrible confusion. Not one single casual user is hurt by VP being expressed as a simple percentage. Not a single one.

Well written.

In the software developer skill sets, there is one aspect that MUST be included - ability to follow test driven development. In an agile - open - face paced model that STEEM blockchain development has, its highly important that the there are adequate tests are included in the code itself. Having enough test cases which get triggered as a part of the CI - CD pipeline is essential.

Going forward I feel the witnesses should also contribute to establishing the CI - CD infrastructure and ensure testing beyond what is possible within the limitations of TESTNET. While changing the code Chainabse code or the DPOS code requires deep knowledge of the blockchains, BOOST libraries etc accepting open source contributions (PRs) may not be easy. But tests is an area IMHO developers can contribute.


Going forward I feel the witnesses should also contribute to establishing the CI - CD infrastructure

I agree. This is a positive overlap, witnesses - because of their extensive use of the platform at many levels - may be aware of some use cases that developers may not see immediately.

Nice suggestion...
I don't know how top 20 witnesses are going to to react to 1st suggestion... But a good suggestion nevertheless.
And 2nd suggestion is most viable... Test net is exegently required.

Posted using Partiko Android

Your post is a wake-up call to the devs. I hope they swallow their pride and take note of this. It is time for me to vote you as witness. I should have done it before I got my spine operation last June 2017.

I agree.
Seriously, I stand behind all your points.
If 15 minutes would be up already, you would have received an upvote.

The way things are done here needs to change.
Stinc want serious business for their upcoming SMT launch in March next year?
Not with that cluster fuck of management they have.

And HF21 (SMTs) already in march?
Has anyone seen a deployment plan for it?
Or a Roadmap for HF22?

Most of my post HF20 rants are not about content, they are about lack of processes and structure.

Better management is needed.
T20 witnesses need to be exchanged, as they all agree in the way things are handled.
They all had the 0.20.2 version running.
Blind trust doesn't work in a public blockchain.
But still not a single post against the way things are working.
I guess they all love their votes from freedom.

Thanks for great post @dragosroua. I am simply in the user community so I will allow other more qualified folks to express opinions.

Hoping that the gradual changes resolves the problems of HF20

What about an idea that the number of witness vote would be decreased (let's say to 15 but essentially less than the number of core witnesses) instead of doubling the number of core witnesses? I can see that you highlight the issue that there may not be enough witnesses to meet a balanced state. Though, I was wondering if that could have a similar effect to keep witnesses who can review code changes and ensure that there are no major bugs.


The rationale behind increasing the core witnesses pool is that the lower the number, the more vulnerable the system is. Or, in software security terms, the bigger the attack surface is.

If we have more core witnesses, then it will be more difficult to bribe them all, or for all of them to be negligent, etc...

Congratulations! Your post has been selected as a daily Steemit truffle! It is listed on rank 8 of all contributions awarded today. You can find the TOP DAILY TRUFFLE PICKS HERE.

I upvoted your contribution because to my mind your post is at least 9 SBD worth and should receive 139 votes. It's now up to the lovely Steemit community to make this come true.

I am TrufflePig, an Artificial Intelligence Bot that helps minnows and content curators using Machine Learning. If you are curious how I select content, you can find an explanation here!

Have a nice day and sincerely yours,

This post is supported by $0.27 @tipU upvote funded by @cardboard :)
@tipU voting service guide | STEEM Monsters Lottery | For investors.

After reading this article, my respect for you has scaled new height Sir.
Such a fantastic article and very very elegant and true insight of what happened. I wish your message and this article should reach those top 20 and also steemit INC. I do also believe that after 2 years of the journey if we dont have testnet and also the awareness and conscousness level of the top 20 is not that par that means steemit Inc is also running short of good developers and time has come for steemit Inc to hire and recruit quality developers otherwise everything going forward will be a nightmare.

I definitely agree that there should be more 'top' witnesses, and mostly to spread out influence within the ecosystem a bit more. There are numerous witnesses in the top 21->40 that have the best interests of the ecosystem in mind.

Though, I am skeptical of:

  • ever having the increase from 20 to 40 implemented;
  • avoiding the same 'witness domination' from the same high SP accounts, even if it were implemented.

Great points at a good next step to avoiding this once again. The increase of witnesses will also have additional positive considerations as it can become more decentralized and the redundancies created can improve the security of the protocol. I would support that. Thanks for your insight and thoughts! We need more witnesses open uo to thoughts like these!

Very cogent explanation of this new development, I believe it has helped some of us a lot to understand the new modarities and the way it is going to work.

Hello @dragosroua, I read your post regularly and every time a little smarter. I thank you for this. I also write code, although outside the blockchain and know from experience that despite hundreds of tests it can always cause problems. That's why I like to repeat your words: "we need to test stuff before we release it."

That's why active and with proper knowledge about coding is required being a witness is not that easy to run this platform they are the backbone of this blockchain

Nice post!

Posted using Partiko Android