PSA: Time to Set Your Witness Server Shared Memory File Greater Than 12GB!

in #psa7 years ago (edited)


( screencap from https://steemit.chat/channel/witness-blocks )

Laid down for a bit of a nap earlier and woke up to my main witness server overflowing it's shared memory file... Causing me to miss nearly as many blocks as I'd missed in over a year of running a witness campaign. My apologies for causing any lag on the network!

Of course my witness failover script shit itself and was unable to switch over my witness keys to my backup (which thankfully was running 16GB shared memory file) and thus I missed 77 blocks.. Unacceptable, will be getting my failover script working before I sleep again. Serves me right for sleeping! What they don't tell you about being a witness is that you are ALWAYS on call, given had I not been napping I'd been able to rectify this issue earlier..

Check out my new high score.... fml


( screencap from https://steemdb.com/witness/misses )

Dropped from rank 7 to 12 it looks like, which is to be expected..! Generally speaking if a top 20 witness is failing at his block producing job it's up to the community to put him in a backup position, no hard feelings towards anyone who unvoted my witness during that overflow blunder! It's our duty to keep the networks blocks flowing I say!

Setting Your Shared Memory File Size

By default steemd sets the shared memory file to 32-56GB so for most this won't be an issue. However some witness server operators like myself choose to run their shared memory file in the systems RAM as shown in @abit's post Best Practice Running steemd v0.16.0 which causes the witness server to overflow with a 12GB shared memory file size set in the config.

config.ini file is located in the witness_node_data_dir directory.

Modify the shared-file-size parameter value to 16GB or more to ensure that you don't end up overflowing and missing blocks! While this will suffice for now it's almost guaranteed that in the future this shared memory memory file size may need to be increased!

I'm off to go write a NodeJS failover script to prevent this from happening again. Was previously using @jesta's failove script seen here but it seems either user error or outdated dependencies may have caused it to fail. I'll open source my failover script once completed.

Have a gooder everyone!


VOTE
KLYE

for Witness!
Sort:  

We’ve (and by we I mean @justinw) found in production that a raid1/raid0 of fast NVMe SSDs is sufficient for running the shared memory file, so that buys a lot of headroom versus trying to fit it all into system ram via ramdisk or tmpfs or ramfs.

Bumping the shared memory file size used to be a regular occurrence for us; with this setup we can set it to something huge and should be fine for quite a long time.

Thanks for the info @sneak! Very good to know for the future. My server boxes have a wee bit of headroom left on them yet but I'll certainly be looking into a setup as described above for future witness infrastructure upgrades.

i dont understand this wizness thing yet.

post dan nice advice @klye.

it's useful

Wow amazing...

I am glad my file is at 24GB, still a lot of room to grow. All the witnesses who just started with a 16GB ram server will pretty soon have a bad time I guess...

Reminds me of South Park :D

Thank you.. I wish I could set it higher then 12... see my eroor error today..;-( https://steemit.com/witness/@rival/step-5-of-becoming-a-steemit-witness-errors-errors-i-hope-i-can

Coin Marketplace

STEEM 0.19
TRX 0.15
JST 0.029
BTC 62676.37
ETH 2581.43
USDT 1.00
SBD 2.72