Witness Monitoring Bash Script (Update 2)
I've been working on some improvements to my witness monitoring script which I first introduced here and then updated here. The script relied on the log output which has gotten a bit chatty since HF20. I started testing out v0.20.6rc1, and it also has some resource credit related log output which can lead to false positives if the logs are parsed for increasing block numbers (as the previous script did). I know you can adjust the logging levels if needed, but I also like to see what's going on there.
The updated script now uses the JSON RPC to get the last_irreversible_block_num
directly and ensure it's increasing. The nice thing about this approach is it would also alert you if there was a consensus issue on chain, even if your node is working correctly. I also added an improvement to resend the text message every thirty minutes. Previously, if the notification_sent.txt file existed, it would just exit without another notice. This approach also cleans up after itself as soon as the node is working properly.
This script does require the json_rpc plugin, but with the new resource credit system, I think it's helpful to have that turned on anyway so you can monitor things there (as there is no cli_wallet support for that data). Using Steem-in-a-box, you can keep those RPC ports unexposed to even the host machine, so they can only be used within the docker image.
If you were using my script before and noticing false-positive notifications, this update may be of use to you.
#!/usr/bin/env bash
if [ ! -f last_block.txt ]; then
echo "0" > last_block.txt
fi
if [ ! -f skip_count.txt ]; then
echo "0" > skip_count.txt
fi
send_notice=false
last_saved_block=$(< last_block.txt)
skip_count=$(< skip_count.txt)
source .env
# Note: if running without cron, add in -it after exec
latest_block=$(docker exec $DOCKER_NAME curl -s http://localhost:8090 -d '{"jsonrpc":"2.0", "method":"condenser_api.get_dynamic_global_properties", "params":[], "id":1}' | jq '.result.last_irreversible_block_num')
# to test, uncomment this next line:
#latest_block="failtest"
if (( latest_block > last_saved_block )); then
echo "We're good. Latest block is... $latest_block"
echo $latest_block > last_block.txt
echo "0" > skip_count.txt
if [ -f notification_sent.txt ]; then
rm notification_sent.txt
fi
exit 1
fi
if [ -f notification_sent.txt ]; then
send_notice=false
skip_count=$((skip_count+1))
echo "$skip_count" > skip_count.txt
echo "Skip count modulus: $(($skip_count % 30))"
# Send a text every 30 minutes
if (( $skip_count % 30 == 0 )); then
send_notice=true
fi
else
send_notice=true
echo -e "Something went wrong. Here's what we got for the latest block: |$latest_block| \r\n" > notification_sent.txt
latest_log_entry=$(docker logs --tail=10 $DOCKER_NAME)
echo $latest_log_entry >> notification_sent.txt
fi
if $send_notice; then
date >> logs.txt
cat notification_sent.txt >> logs.txt
echo "There seems to be a problem."
echo "Last saved block: $last_saved_block"
echo "Latest block: $latest_block"
python sendnotice.py
fi
Luke Stokes is a father, husband, programmer, STEEM witness, DAC launcher, and voluntaryist who wants to help create a world we all want to live in. Learn about cryptocurrency at UnderstandingBlockchainFreedom.com
Great, congratulations.
very good post
Posted using Partiko Android
Well work Luke. Its good to see your Steem update