State of the Witness #7 - Stuck in the Woods, No Disk Space, Lessons Learned

last month

state-of-witness-banner-7.png


Hey Steemians. I had to do some land surveying work in the woods today and THE WORLD LITERALLY ENDED.

LITERALLY!


Source

Well, maybe not literally. My witness server experienced a hiccup during a routine weekly update that caused logrotate on my server to quit functioning. This, along with AN EVER LOVING SHIT LOAD OF ASCII CATS (eyes @abit), then caused my main hard disk to fill with trash logs that would normally be properly managed and only really available if debugging is necessary.

Of course, no disk space = programs eat shit. They don't work right, if even at all.

All of the problems have been corrected and the server is replaying the blockchain. Witness operations will resume AS SOON AS POSSIBLE.

Lessons Learned

Going forward, I will be setting up multiple monitoring systems for basic computing resources that need to be managed to have steemd constantly run without issue. On top of this, I have already began creating an ansible playbook that will be available for anyone to use that will allow a COMPLETE witness server setup from creating server to witnessing blocks with minimal alteration to a single config file. I am mainly making the playbook to use in my own automation.

Vote for @netuoso as Witness

(This is a big one because I lost a few votes since I was unable to return to my server and perform the necessary disable + block replay as fast as I would have liked; and your support makes me smile, of course)

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  trending

Resteemed. I support you!

Supported. Please consider supporting my friend @themarkymark .

I feel your pain. Last time that kind of thing happened, I had a crash course on setting up zenoss and automating fixes for every way I could think of it going wrong again.

Have what @crimsonclad calls a "spite vote" man. Just cause I don't hate you enough to move my slider down for it.

·
·
·

Many spite, much copious, very vote.

@netuoso very informative post sir thanks have a good day ahead

Nice to meet you, @holl.boll
Wow, this is somehow a dream come true after all these years envisioning it coming to our everyday world.
I votes you post plase vote my post thank you
Thanks a lot for sharing, great job as always,

It happens to the best of us. You have been kicking ass if you learned something you came out ahead.

Check out NetData, it's a really good server monitor similar to New Relic but open source.

·

I'll check it out for sure. Any tools I can use to help manage things are always good.

·
·

It's really slick, and you can make custom hooks for it. I'm thinking about making some for my bots and witness.

Glad you were able to fix. Question for you, What happens when you miss that many blocks. Is it given to the next witness?

·

Yeah. The block I was scheduled for counted as a miss to my witness and moved to the next scheduled witness.

·
·

for your honesty and transparency I give you my vote. :) Mistakes happen, being open and honest about it make you human and I appreciate your efforts to correct the situation and prevent it from reoccurring.

Man that sucks, but when it rains it poors. Like the pilot told the technician when he filled up his plane with diesel instead of jet fuel. I want you to fill up my plane everytime now because I know you wont make that mistake again.

been there, done that! Wish you all the best

No need for the box, you are in the top 50 now, you got my vote.

You seem like a man who gives a fuck. I respect that

Oh man, I did a check and only one other besides yourself was False! How can that be as they were listed on top 50?? Now what?