Missing 3 Blocks due to Network PDU Failure and Auto Switch to Backup Node
Last night, about 2:00AM (BST) in the morning of 9-Oct-2020, the node that was producing the blocks started to miss 3 blocks in a row
and then have triggered the auto-switch to one of the backup node - and I got a call via IFTTT as a on-call notification shortly afterwards.
Checking in the morning on the node to find out what happened but seems cannot SSH:
Cut a ticket to support and got:
Upon further investigation, the server was powered off which was an issue due the the PDU port it was connected to. We have now moved the server to another PDU. We are showing that the server is powering on. Please allow some time for the server to boot.
Then, when the server is rebooted, the steemd were broken:
And from the log, it showed the incident happened at block #47560386
Then, I have to recovered it by a backup that was made 10 days ago:
And re-synced within an hour or two:
Lessons learned: always have a backup witness node!
Find Cheap & Bargin VPS: VPS Database
Follow me for topics of Algorithms, Blockchain and Cloud.
I am @justyy - a Steem Witness
https://steemyy.com
My contributions
- Steem Blockchain Tools
- Computing & Technology
- Download Youtube Video
- Find Cheap & Bargin VPS: VPS Database
- Online Software and Tools
Support me
If you like my work, please:
- Buy Me a Coffee, Thanks!
- Become my Sponsor, Thanks!
- Voting for me:
https://steemit.com/~witnesses type in justyy and click VOTE
Alternatively, you could proxy to me if you are too lazy to vote! and you can also vote me at the tool I made: https://steemyy.com/witness-voting/?witness=justyy