"Steem Is Broken!?!" A Conversation with Sida of Partiko
This has been a challenging couple of weeks for the Steem blockchain. We had a chain freeze recently and Hard Fork 20 which just went live had some challenges with the Resource Credit system (along with some bugs) which really impacted the user performance and has taken a solid day to fix due to unavoidable node replay times. In this conversation we touch on a lot of the challenges we face with decentralized systems vs. centralized systems and why resetting our expectations is so important. We talk about the coordination problem and the costs associated with getting many independent actors to agree to forward action. We go over lessons learned and what can be brought over as best practices from the existing, centralized software development industry. We also touch on some things Steemit, Inc. could have done better and some things they are doing right such as supporting a testnet (which witnesses and application developers could have used more actively than they did).
This conversation was mostly about being heard. As someone who built my own software as a service company for over 10 years, I understand how frustrating it is for software not to work as expected. I understand why Sida and his users were frustrated and why many other application developers and community members are frustrated also. Part of this involves resetting our expectations to better fit the realities of decentralized, blockchain systems. This is cutting edge technology, and we are all pioneers blazing new trails.
It was a fun conversation, and I hope you found it useful also. Please share it around to those who may also be frustrated with decentralized systems not working according to their centralized expectations.
Links mentioned:
Eating Your Own Dog Food: Is the Steemit, Inc Team Regularly Using Steemit?
https://steemit.com/steemit/@lukestokes/eating-your-own-dogfood-is-the-steemit-inc-team-regularly-using-steemit
Witness Update: 15 Hours In Front of a Computer in Puerto Rico
https://steemit.com/witness-category/@lukestokes/witness-update-15-hours-in-front-of-a-computer-in-puerto-rico
Hardfork 20: What to Expect Tomorrow
https://steemit.com/steem/@steemitblog/hardfork-20-what-to-expect-tomorrow
Video: some thoughts on the DLive exit and the Hard Fork 20 resource credit system
https://steemit.com/dlive/@lukestokes/video-some-thoughts-on-the-dlive-exit-and-the-hard-fork-20-resource-credit-system
To fork or not to fork? Is that even a question?
https://steemit.com/witness-category/@lukestokes.mhth/to-fork-or-not-to-fork-is-that-even-a-question
Pull Request Submitted for Condenser
https://twitter.com/lukestokes/status/1044687128456822785
Partiko
https://partiko.app/
https://steemit.com/@partiko
Cutting Edge Technology requires Cutting Edge Developers. There clearly is a sizable incongruency between the two. And the current, less than fair quality of HF20 execution with such egregious oversight of not adding an RC reset scaling speaks for itself as to exactly which part is out of sync. What has clearly been lacking is an effective "de-risking" strategy to this HF20 launch! In other words, not enough "what if..." scenarios were covered; the basics of an effective Project Management and the common cause of "oversights"
Front END of STEEM is paramount and yet all emphasis keeps stoking the back end. There is an incongruency of resource allocation such that the Front End is SUFFERING for improvements! In this case, Steemians need not reset expectations but rather raise them. This is where competition comes in to provide exactly what users want and better it. At the end of the day, users always get their ways.
Remember, even in a decentralized environment, STEEMIT must be a USER pull and NOT a DEVELOPER push process. A Developer Push often misses the mark in meeting the USER needs and wants. This HF20 has clearly forgotten this such that the best anyone can now expect is hope for the five days to quickly pass and this ignominious event be swept into the dustbin of history.
I'm surprised to have such a controversial figure commenting here. While I have your attention, would you mind discussing any relationship (if any) you have with @ranchorelaxo and any concerns within the community about the disproportionate amount of the rewards pool your analysis posts accumulate on a daily basis? You've been at the center of a lot of drama, flag wars, etc. I'd be interested in hearing your side of the story. Maybe do an interview, if you're up for it. If you have a post that summarizes things, please link to it in a reply. My hope is Steem will be mostly for people who care about wealth creation over value extraction and consider the benefits of psychological egoism.
As to your points, there are many front ends to Steem. As I've said before, Steem is not Steemit. Steem is more valuable than Steemit. Steemit is, in many ways, just a reference implementation of what's possible with the Steem blockchain. We can use https://steempeak.com/ or https://busy.org/ or build our own, if we like. Saying the front end of Steem is paramount doesn't make it so and may not align with the long term goals of Steemit, Inc. I talked about this a bit after SteemFest²: Bridging the Gap Between Perception and Reality.
I agree, HF20 has been pretty frustrating. A post telling everyone to expect some disruptions is quite different from "things will be completely unusable for a time." It seems we have a new update which will take another day or so to replay the full nodes to see the benefit of. Witnesses are already building and replaying their backups to be ready, but replay time is unavoidable at this point. HF20 was rough, no doubt. To say it was a failure would require us to compare it to similar technologies in the blockchain space which have done better. Are there any examples or is Steem doing something brand new?
You mention competition. What are some of your favorites at this point? All I've seen, so far, are unfulfilled promises and/or centralized solutions such as wordpress with token incentives added on top.
I think HF20 was a pretty rough release, for sure. I hope we get a detailed explanation of what went wrong from the team that built the code so we can learn from it and create better expectations in the future.
What a great Interview and a great dialogue with a “controversial” figure as you put it.
I hope that he takes you up on your offer.
Looks like steem is almost fully functional again. Great job by the devs working on the fork! :)
This discussion answers many of the questions people have been asking. I tweeted it! I know that people don't want to spend 47 minutes to learn the nuanced details of the situation; it's easier and feels better to just vent frustrations. Fine. But for those who really want to know what is happening, I'll tell you that Steem is a new system in beta and it will not fit into the box in your mind that you want it to be in. Learn about the new box that it fits in, and in my opinion, is doing well in.
Thanks Kenny. These are complicated systems with complicated solutions. People have centralized expectations on decentralized systems. It does not map well.
So far the interview looks great and I wish I had time to listen to it all. With respect to the issue of reverting, I just wanted to highlight the point that "reverting" would be intentionally choosing a broken system. It would be like in the midst of a life saving surgery someone went, "Wait, this hurts, I want to go back to having my fatal disease." Could we have done a better job facilitating the transition? Absolutely. But this is totally new territory and there are no guides to follow. This change in particular was an incredibly critical component of the stack. It's really like swapping out the engine of a car while in motion, as opposed to replacing the rear view mirror or the radio. Every hardfork will involve a different piece of the stack, but I suspect few will involve such a foundational component.
We just have to keep getting better as we go. Ultimately, the main question behind hardforks will be, "Is the potential improvement that comes from this code worth the risks associated with changing the blockchain?" There will always be risks associated with hardforking. Blockchains aren't optimized to be changed, precisely the opposite. But Steem's ability to make these improvements is, IMO, a critical component of its value proposition and those who think that by simply freezing progress we will somehow benefit the system, I believe are missing this critical point. What we need to improve is how we work together as a community to develop and test the code to ensure risks are minimized. There's a lot we at Steemit need to do to that end, but there is also more that the witnesses and other developers need to do as well. But I have never been more confident that we will all work together to do what is necessary. These are trials by fire and what doesn't kill us makes us stronger.
Thanks @andrarchy. Maybe when you get bored in all your free time 😂you could put it up on 2x play seed on YouTube (life saver!) to catch the rest. We do address how the witnesses and the application owners can and should do more to help with the testing process. More time doesn't automatically mean people will prioritize testing. I think this experience showed us all how, if we want a better outcome, we need to take more personal responsibility for improving the process. If the testnet wasn't exactly right, we should have figured that out early on, made changes, and then moved forward with more accurate testing on the testnet. As it was, people (myself included) only got around to really tinkering at the last minute. We can't do that going forward if we expect a different result.
Amen. And just to clarify, I wasn't pointing a finger at anyone. While this has all been less than ideal and extremely stressful, I can't help but feel proud of what we've all managed to accomplish, even if to some it seems like an apparent failure. We just improved our bandwidth algorithm. So far every issue is temporary. I am perfectly comfortable saying that while we did a bunch of things wrong, we all have plenty to be proud of.
I do think Steem is an amazing blockchain doing some amazing things which only some have the technical capacity to fully appreciate. I do think there is (and will be) plenty to be proud of. I also hope there will be a detailed explanation of the things which did go wrong so the community can better understand the quality of the developers building Steem and what steps they are taking to avoid these types of mistakes in the future.
I would certainly be more frustrated if, after each hard-fork, I had to spend a tonne of mental energy figuring out which chain is the most adopted to move forward with, or start juggling multiple wallets. I appreciate that there is continuity in the community and core-chain (for lack of a better term) rather than chaos that comes with splitting chains and issues of that ilk.
If this were similar to ETH & ETH-Classic, or Bitcoin and Bitcoin-Cash -- I'd have a SUBSTANTIALLY more difficult time convincing my peers to give it a shot. A little downtime and hiccups in early stages like this are significantly more palatable, IMO.
Good to hear some reassuring words that I am sure will calm a lot of nerves.
I am stoked that I can use Steem without 1$ invested (just my time though) and that this is an ever evolving process. That we are all part of the ride and this is just the beginning of something far greater.
I would love to see a Steemit handle set up as a platform. Where all the top 20 witnesses have their posts about the upcoming hardfork resteemed. Not sure if it would be for everyone, but I would be interested in understanding various perspectives. Then I could inform myself from it and vote for a particular witness for the HF consensus.
Yes, we all need a good festival every so often :) I love how energetic Steem feels and is driven by passion. Even though we have our "real" lives to get on with. We spend so much time on here.
Thank you for your dedication and time,
You coming to Steemfest3? That's a fantastic way to see all this energy in the physical space. Highly recommended. :)
As for staying informed about what the witnesses are up to, check out @timcliff and his weekly reports of the witnesses he publishes on Sunday nights.
I wish I could, but highly unlikely as of now. I am aiming for the next one though :)
I will check out the weekly report and thank you for the source.
Oh, would it be a good to time to get someone full time on all this stuff? Like you mentioned a while back. I would be still keen to weekly contribute (SBD/Steem) in any way I could to that effort. A community effort should make this relatively easy to happen and will benefit us all.
I hear what you are saying about people having centralised expectations of a decentralised network...
But...
If the steem blockchain is two years old (having started in 2016), and it takes a day for the nodes to replay it from the beginning, that is a problem.
What happens when the blockchain is ten years old? Will it take five days to replay the nodes? And when the blockchain is twenty years old - will it take ten days to replay the nodes?
And that is with everything going fine. When things go wrong and you have to put in a fix and replay all over again, then even when people are patient and have "decentralised expectations", you have a serious problem.
Nobody is going to wait that long. Which means that either you guys need to stop doing fixes and hardforks, or you change this replay-from-the-start business as it doesn't scale.
Most replays are never noticed. Most bugs don't require immediate replays by all nodes for the system to remain functional. The frozen chain issue we had recently and this issue with the hard fork were extreme examples. Mostly replays are done on backups and then brought in seamlessly (which is why most people don't notice them).
I do hear you, though. Speeding up replays will help minimize downtime in situations like this. It's certainly being talked about and improvements in both the hardware possibilities available and in the software may help make this a reality.
Thank you Luke for taking the time and explaining all of this. Hopefully this is valuable to all Steem users!
Posted using Partiko Android
Thanks for reaching out to me! I really enjoyed the conversation.
It is
Posted using Partiko Android
How can we ensure that blockchain remains sustainable as time passes? If it already takes 24 hours to replay a node does that mean it's going to take two days in another 2 years? Can we create checkpoints that move up the genesis block to a more recent date?
Spoke too soon: This issue is talked about at 39:00
Genesis blocks don't generally "move" but ideas are certainly being thrown around. Usually, replays are never noticed by the community because they are not done all at the same time due to a critical bug. In this case, it impacted everyone so all nodes had to be replayed.
There's also the idea to consider that computers are getting faster and cheaper. In two years there may be other hardware level solutions which speed up the process.
It's certainly something we'll have to get serious about in the future.
I didn't watch the video, too long and my attention span is very limited. What I want to say is that this hardfork's outcome up to now has been very disappointing, you would think that with a technical team available and with all the time taken to make this hardfork any glitches would have been foreseen, to me it is either incompetence (which I highly doubt) or it is the outcome that was wanted. It is kind of sad to see this glitch affectign nearly all users except the big guys.
Watch the video, and you may come away with a broader perspective.
Or keep your current opinions.
It's up to you.
I'll look forward to listening later.
At the end of the day I try and keep the 'beta' logo in mind at all times, and the fact that many people on here aren't salaried as such.
Crucial to be realistic!
Posted using Partiko Android
I believe we learn more when problem occur and adapt along the way. This is part of the life cycle of software development.
Posted using Partiko Android