Steem Network Status - And A Witness Update
Following a small bug in the new version of the steemd daemon for the upcoming HF 20, the Steem network halted yesterday. The reason is highly technical, in layman terms a part of the witnesses were running a certain function, while the others didn't, so at some point a critical percentage was hit, a fork occured and witnesses stopped to produce blocks.
Both Steemit Inc developers and some prominent witnesses (@pfunk, @someguy123, @smooth and @abit are the first coming to my mind, but there were more in the witness channel of the steem chat, so a big shout out to all of you guys) started a bug hunt which lasted a few good hours.
Eventually, a decision has been made to roll back to 0.19.12 version on the entire network, until the HF 20 version is correctly patched. That's exactly what's happening right now.
My witness nodes are both replaying the blockchain version 0.19.12, and they are at about 40% both. Until they are fully synched I disabled my witness node to not cause even more lost blocks. I expect them to be fully functional in about 7-8 hours from now.
Again, kudoz to all involved in this incident for acting fast and putting things back on track.
I'm a serial entrepreneur, blogger and ultrarunner. You can find me mainly on my blog at Dragos Roua where I write about productivity, business, relationships and running. Here on Steemit you may stay updated by following me @dragosroua.
Wanna know when you're getting paid?
|
I know the feeling. That's why I created steem.supply, an easy to use and accurate tool for calculating your Steemit rewards |
Now i understand whatvwas going on yesterday
Great info!
I was wondering why the nodes were disabled.
I wonder why smooth is running 19.5.
https://steemd.com/witnesses
I was wondering what the problem with the blockchain was. I feel these hiccups need to be communicated in some way to the community as they happen because they are kind of nerve-racking.
It was actually communicates in many discord channel by many witness, I know sir cock and many others, they were communicating the progress as its going on.
Some of us don't use discord and would rather read an official post by the devs on some back-up platform like medium for example.
Nice job managing your nodes and giving status updates after yesterday.
Seems voting for you as a witness a few days ago was a good decision.
Keep up the good work!
Posted using Partiko Android
Thanks for your witness vote, appreciated!
Your welcome!
Posted using Partiko Android
Glad steemit is back and hope hg20 will work perfectly after bug resolved.
Thanks for a clear explanation of what happened, and what to expect. I really do appreciate that.
These first two commenters are obviously reward pool hunters. They voted for their comments and not your post. That really irritates me.
Hey, @dragosroua.
I understand the desire to keep a positive tone here—rarely anything good is accomplished through negativity—as long as we're also keeping it real.
I don't think it's being overly negative to state that the blockchain being down for 10-11 hours in my case is really that okay. So, it would be nice, not by you, but someone—Steemit Inc through the steemitblog—would come up with a way to keep the blockchain running particularly in times like this, but any time, really, and announce it, and then implement it.
I understand that the blockchain programming is hefty, and that bugs are bound to happen and to be missed. Can bug searches only be done manually? I would hope not. I would hope that some kind of app could be constantly looking for bugs in the background (maybe it already is), or some kind of discrepancy in the coding between forks even.
I would also hope that before all 20 Top Witnesses committed to the new hard fork, that the older version could continue to roll on somehow until it was apparent that the new fork was okay and going to take. It seems like there would be a way for that to happen.
I can be thankful that my account is okay and that things are back up and that I can now leave this comment—and I am. But I also feel like it shouldn't go down in the first place, especially before anything permanent or however you say it is supposed to happen.
Bugs are rarely occurring because of software itself (hence an app monitoring the blockchain for that is not plausible) but because of humans. With that being said I can assure you that the steemd blockchain is coded at the highest standards and it has functional and unit tests in place. Each version and each release comes with these tests, which are basically ran to maintain the integrity of the software itself between various upgrades.
But the nature of blockchain itself makes it difficult to test it. You may end up with a case in which all the functional and unit tests are passing, but some edge case on the real blockchain will not. In this specific case, the bug was triggered by somebody issuing a vote on the lockout period, which, for some nodes was 12 hours, but for others 12 seconds. To put in practice such a test is really time consuming and really hard to implement. I salute the fact that there is a Steem testnet and, in an ideal world, this bug should have been triggered there. But again, the nature of the blockchain makes it very hard to do so. Testnet doesn't have the same number of witnesses or even the same structure of them.
I am also annoyed about these hiccups, but, as a programmer doing this thing for a living, I know they are inevitable. Sooner or later a big boo-boo will happen. And when it happens, the only thing that matters is how fast and dedicated (and lucid) those involved in fixing it are.
And in this case they did their job very, very well.
I appreciate all of that. I've been told enough times of the complexity of the blockchain and the inevitability of a bug. And while that may very well be the case today, as more and more people gravitate to any blockchain of any kind, and as entire industries are built upon them, it is my hope that someone, somehow, someway, will figure out how to keep the blockchain running properly and accurately—without causing greater issues as it continues to run—and that it won't be such a difficult thing. I imagine money and resources will help that come to be. And I guess for the meantime, we hope it won't happen again and we all find something else to do when it does.
Job well done to You guys, thanks for not letting us down.
I truly believe that in anything you choose to do there is always room for improvement. No matter how good you can get at something, there will be always something that you can do to improve yourself.
From this perspective, I agree there is room for improvement in how these type of situations were handled by the people in charge (being them part of Steemit Inc or just witnesses, like me).
On the other hand, I choose to focus on the positive. It just gives a better quality of life, you know...
Yeah they have done a pretty good job... A shout out to all who were involved...
Posted using Partiko Android