Why did STEEM blockchain froze ?
Apparently someone initiated a transfer ? of negative VESTS (VESTING SHARES)
THERE IS NOTHING TO BE PANICED ABOUT
Once the chain comes back, we should be able to see the transactions and get more details.
the details of the fix:
We can see the following changes in the code and an account nijeah
auto account = find< account_object, by_name >( "nijeah" ); if( account != nullptr && account->to_withdraw < 0 ) { auto session = start_undo_session( true ); modify( *account, []( account_object& a ) { a.to_withdraw = 0; a.next_vesting_withdrawal = fc::time_point_sec::maximum(); }); session.squash(); }
nijeah
So it seems the account had withdrawn -1,000,000,000,000 VESTS from vesting balance! yea, NEGATIVE!!!!!.
Basically powerdown!
7 days ago Start power down of NaN STEEM
7 days ago Start power down of NaN STEEM
7 days ago Start power down of NaN STEEM
7 days ago Start power down of NaN STEEM
Now, (-1,000,000,000,000 ) will make the total STEEM in supply to negative and thus the blockchain on its knees.
@nijeah vs @haejin
I love this bug for the account which is used to demonstrate the bug.
All hail @haijin and @nijeah :-D
Michael Vandeberg (@vandeberg) : Appreciate your hard work and the fixes. Hats off to you!
.
What's the link for the commit with the fix on GitHub that you screenshot?
Are these invalid transactions going to remain part of the blockchain? More context would be much appreciated. Additionally more information over how this issue was identified and resolved.
https://github.com/steemit/steem/pull/2583
Changes are here : https://github.com/steemit/steem/pull/2583/commits/1197e2f5feb7f76fa137102c26536a3571d8858a
I will explain in detail later. Roughly this does
The check is done immediately after the blockchain file is opened to avoid execution. (roughly. though the execution path is not linear)
It looks like @nijeah executed via JS. I couldn't reproduce this yet as I had to make changes to my environment to run this. Will share the exact errors thrown tomorrow. I think the race condition might have created a transaction time out for the block and the block producers might have went trying to producing the block and into an infinite loop or until all the block producers available were over.
Thanks for the details. It's always fascinating to learn how things break! It seems like some people are accusing @nijeah of being a bad actor, but I'd rather have the kinks worked out sooner rather than later.
So if I understand correctly, @nijeah made 4 offending transactions 7 days ago as per https://steemd.com/@nijeah:
Those should-be-invalid transactions went unnoticed until the first thirteenth of the power down executed (or started crashing nodes upon execution). So those four transactions will remain part of the blockchain, but the new software is hardcoded to ignore them and prevent future negative power downs like this.
I believe Bitcoin's source code also has many of these historical hardcoded artifacts... bugs that because they were on a live network now are forever part of the protocol up to a certain block height.
exactly my thoughts. Any software which has successfully "consumed" more CPU cycles is more refined - like old wine. So is Bitcoin and so is STEEM. :-)
IT seems, the account could belong to netuoso:
2018-01-06, 00:47 netuoso bittrex 43.528 SBD f62cb7ea66e84c94867
2018-01-06, 00:47 nijeah bittrex 26.663 SBD f62cb7ea66e84c94867
they used at least the same bittrex address
no clue - but this is a bug worthy of a big bounty. I wonder whether STEEM has a bounty program.
Is there anything @ned can do about this?
This was actually a very minor mistake with huge impact and for the time being it’s fixed by
preventing existing negative withdrawal attempt
future attempts are also prevented
A better, ie faster fix will be provided in HF20 as per the comments.
This is one of those very unfortunate bugs that exists in all software.
Neither @ned or anyone else can prevent things like this and I would consider this a minor error. I think we must appreciate the long hours the unsung heros - the developers, @vandeberg, code reviewers and DevOps Team has pulled to fix this bug.
A simple "validation" check should fix it. Why it is not implemented in the first place. We are joke due to this programming error.
@yehey - While this could have avoided with a validation error, since the blockchain managed to work without hitting this error itself is a proof that its not major one. This is like a rocket launch failing and I think we all should realize that the software systems are prone to fail no matter how much quality assurance is done. While its bad that this happened, I think we should look the ability of the team to come up with a working patch with only minor chain base version mismatch in such a short time.
To Err is common but to fix them so quickly desrves some good words in IMHO
All credit to @vandeberg --> https://github.com/steemit/steem/pull/2583
I consider it MAJOR if the blockchain stopped working for hours, don't you think?
We can't sugar coat this incident. I understand you but in the world of business this is really bad.
yea, in the world of business, its a serious one :-(
What is negative vesting withdrawal? Can you elaborate?
A simple way to explain it. Withdrawing without money.
Essentially it will try to withdraw the VESTED SHARES but with a catch ---> it will be a negative number. theoretically this negative withdrawl has evaporated all the STEEM in circulation which is 270 million or so tokens but due to checks like VESTES being with other users etc will not happen. So I think first it might have reduced the entire VESTED Balance of @nijeah if there was any and then it will try to find further VESTED shares. I think this will create a "race condition" and the block production will stop as this particular operation will never be completed. Since the operation was a valid operation, the blockchain and thus all the witnesses will try to wait until the operation is completed.
I think this is what might have happened. Trying to re-produce this now on a node with 1 witness and stable production. Will update once its verified.
so was it an intentional attack or just someone trying to pull a scam or just an accident?
looks like someone behind @nijeah tested a negative vesting withdrawal and it worked. Then they verified and performed 2 big withdrawals (powerdowns) which will create a race condition one week (7 days VESTING_PERIOD ).
I don't think it was a scam or an attack -- I would say this is a curious one :-)
isn't a negative withdrawal essentially an attempted deposit?
This post has been rewarded with 100% upvote from @indiaunited Community account. We are happy to have you as one of the valuable member of the community.
If you would like to delegate to @IndiaUnited you can do so by clicking on the following links: 5SP, 10SP, 15SP, 20SP 25SP, 50SP, 100SP, 250SP. Be sure to leave at least 50SP undelegated on your account.
Please contribute to the community by upvoting this comment and posts made by @indiaunited.
nijeah! or haejin's proxy?
From the other side of blockchain universe and thus anything and everything he does is in negative :-D
But sometimes good to shed darkness on the light.
Thanks for this - reminds me of when I found the "bandwidth error bug" many months ago. That too was triggered by an "anomaly".
yea, its good that the bug was identified. Though I wish if the person had raised a redflag & reported the bug
How did you find out this :)
Well I am using STEEM code base for the project I am working on so code changes are closely monitored....
Proof that Steemit is still in "beta" testing.
I was wondering what was happening. Now I know.