think I should probably be doing these more often but the process of learning things often involves lots of mistakes and consequently one tends to think at this stage in the process, while getting everything full stable and reliable, that there is not much to tell. It's not impressive.

Of course stuff happens, but it's all irritating things and frustration. When my server misses a block, it's at least 8 hours before I can confirm that everything has worked that I have done to fix it, even if it all looks correct and the logs are churning every 3 seconds.

So, I will just give a short summary of what I have been doing.

steemd is a pig for disk access. Most VPS hosts sell rubbish with throttled hard disk access

First two months of my witness operating was terrible. I spent way too much money on my server, and it had way too slow disk performance, which led to all kinds of problems. steemd works with a very large file, typically 20Gb for a witness, but the code is written as though this disk is memory.

This is why many other witnesses are running their witness on /dev/shm the linux ramdisk. Basically there is no caching strategy built into the code, what you get is whatever is default configured in the kernel as to how often it can block the process while writing. If a witness node is in a blocking disk write, and gets the call to do a block, it will often miss it.

With my current life situation, where I have to spend 2-3 hours a day offline in transit, and work in environments with people who are drinking too much coffee to kill the boredom, and becoming boisterous (I fucking hate the way people get so loud when they drink more than a couple of coffees), and often drunk people around me, I can't be futzing about with fiddly stuff like maybe someone with a daily routine and a house and a quiet place to do administration work. I need simple and reliable, and not so difficult to get running.

The worst aspect of using ramdisks is that if the system loses power or you need to reboot it, the blockchain has to be replayed again. I only had one server running until last week, so I would often have no witness up because it was replaying.

The new, shiny fast SSD server (s)

But, as some may know from reading or who talk to me, about a month ago @liondani put me onto a much better hosting service. I pay far less now, which is kinda important, but more important is that for my less money I have a very fast secondary server that I use for other things.

It's now running the backup, with a script that toggles the key so the backup makes blocks) and I had a website running, but I haven't got to re-installing it after I broke my previous system trying to install a mail server (you would not believe how difficult mail server setup actually is). I will be getting the mail server, and website back up and running shortly but with HF17 coming up and me in favour of it, I had to make sure my nodes made a block before today, later today the HF 'vote' goes, I want my entry here: https://steemd.com/witnesses to show 17 again (I had it installed 2 weeks ago but @smooth said there was a bug that HF would trigger and be careful).

I don't think I count in the veto process but 17 is more stable and faster so I will run the latest code anyway.

New primary server

I now have also a smaller primary node configured, it has 12Gb memory and fast SSD, 100Mbit line. In the last few days, after the mail server fiasco (which I am loathe to talk about, but people want to know!), I have readjusted my policies and changed how I am running the servers.

Docker and Arch Linux

Firstly, I haven't yet migrated the primary to Arch, but the secondary, the big fat thing with 50Gb memory 1.2Tb ssd and gigabit connection is now running Arch Linux. Arch Linux has a much more efficient package manager, generally runs faster, and puts even less stuff on by default than Ubuntu Server. But of course, building steemd on it is a nightmare, because of boost, and the incompatibility problems it has with some versions that are in the distribution repository, like even ubuntu 14 you have to manually build boost (which is a swear word in my book).

So what's the solution?

Well, @someguy123, who is a really cool guy, actually, has been developing another docker than the official one from steemit, and I was using it but it didn't suit the way I wanted to do things, so I have been learning how to work with docker and building my own system to work with it. Docker gets rid of all that nonsense about compatibility and dependencies, you literally can even take a docker image and run it on a Mac or on Windows, same as on Linux.

Docker basically lets you set up a minimal ubuntu 16.04 system, the one that works best with steemd, install boost from the repository (much faster and simpler) and then some fiddling to configure how you launch it, and voila! It works!

But it took me a few days to fully understand how to set it up how I wanted it. For those who understand Docker, I did not want to be stuck with the usual methods that make the container hard to get into. I created a small script full of shell aliases, and wrote my own custom Dockerfile, and one of my aliased commands starts up steemd exactly as though it was built inside the main system instead of in a container. The data directory is accessible and is overlay-mounted inside the container so I don't have to enter the container to edit startup and monitor scripts.

Current State of Affairs

I have now got a primary and secondary, as I mentioned, and I have had to revise my failover script because it kept failing from RPC servers resetting connections, and I couldn't figure out from the piston-lib code how to handle this exception so the infinite checking loop would continue. So I took another approach.

The monitor script is now written in shell script, and here it is:

#!/bin/bash
WIF="witness account active key"
WITNESS="backup witness public key"
URL="server address"
   
while true
do
  if (( `ssh -p 8008  $URL 'echo $(($(date +%s) \
    - $(date +%s -r /path/to/log )))'` > 45 )) || \
    ! ping -c10 $URL
  then
    echo "Last modified date more than 45 seconds ago, or unresponsive to ping - switching to secondary"
    ./switch
    exit 0
  fi  
  sleep 5
done

As you can see, it uses ssh to run a little shell command that basically tells you how many seconds since the last time steemd wrote to the log, or if the server does not respond to ping. It then triggers the key-change that tells the network, and my secondary server, that it's now the active witness, which sits there chugging away in the background keeping the chain up to date but not making blocks, waiting for the public key matching its' signing key to become the active one for my account.

The 'switch' command is from a modified version of my previous python failover toggle script, which all it does is take the active private key and the public key of the backup witness, and broadcast a witness_update transaction that tells my other witness it is now up and presumably this is because this script detected my primary witness went dark.

#!/usr/bin/python3
#
# l0k1's witness toggle switch
#
# Usage:
# 
# WIF="<posting active or master key for witness account>" \
# WITNESS="<public key corresponding to private key on backup witness>" \
# switch 
#

import asyncio
import websockets
import time
import sys
import random
import os
from pistonbase import operations
from collections import OrderedDict
from piston.transactionbuilder import TransactionBuilder
from graphenebase import base58
from piston import Steem

owner = "l0k1"
url = "https://steemit.com/witness-category/@l0k1/witness-update-jan-31-2017"
account_creation_fee = "10.000 STEEM"
maximum_block_size = "65536"
sbd_interest_rate = "0000"
fee = "10.000 STEEM"
block_signing_key = os.environ [ 'WITNESS' ]
st = Steem ( keys = [ os.environ [ 'WIF' ] ] )
 
tx = TransactionBuilder ()
tx.appendOps (
  operations.Witness_update (
    **{ "owner": owner,
        "url": url,
        "block_signing_key": block_signing_key,
        "props": { "account_creation_fee": account_creation_fee,
                   "maximum_block_size": maximum_block_size,
                   "sbd_interest_rate": sbd_interest_rate},
         "fee": fee,
      }
    )
  )
tx.appendSigner ( owner, "active" )
tx.sign()
try:
  tx.broadcast ()
  print ( "Successfully switched to secondary witness" )
except:
  print ( "Transaction broadcast failed - DANGER WILL ROBINSON!" )

I know that most of the people who might want to vote for me probably don't really understand or care that much about this technical stuff but I want my learning process to be logged for my own benefit, and also by publishing these things others can learn from it. As a witness, transparency and accessibility are important values to me.

Closing comments:

I need to yet fully polish up how the servers are running, and I want to migrate the primary to Arch Linux, which will be a day-long process probably, but they are up and running and I have an automatic feed-setter that polls hourly from coinmarketcap to set my feed price. The primary is isolated more than the secondary, and is the only program running on the system, so other than connectivity, power or bugs in the code, it should pretty much run for weeks without a hiccup.

But if something goes wrong, and it stops making log entries, after 45 seconds, it will switch to the secondary. Speaking of that, I need to also add a condition in the monitor script that performs the switch also if the primary's SSH server is also unreachable (connectivity and power outage would likely be the cause then).

I would very much like to be able to rapidly, instantly deploy a whole new configuration for the primary, in case for whatever reason the installation is b0rked, but at least, at this point, I have a neat little folder that I can just copy over to a system with Docker installed. I will continue to refine how it works, by replicating the secondary's docker package folder, and give it new keys so it doesn't think it's on-duty, and when I have fully certified it's working, write some documentation and finally upload it to github. I need to write the documentation for my own benefit! Sometimes I forget what does what, because I made it.

New Supplies

Thanks to the market waking up to Steem (I do wonder whether Dan's resignation was a publicity stunt), I was also able today to get a couple of important supplies, a nice fast USB flash drive so I can do rescue and reinstalls of my laptop, and because of having all kinds of trouble with a huawei e3372h-153 4G/LTE dongle to give me connectivity in the evening, I got a cheap and crappy alcatel phone (with 3G) that only cost the same price as a 4G battery powered hotspot. It will also serve as a way for me to get notifications once I get a working dockerised email service running on my fat secondary server, and a place to back up my work and important things.

Thankyou for reading and thanks to everyone who votes for me. I hope that I can help get the Steem Ship on a one way trip to the Moon :)

---

We can't code here! This is Whale country!

Vote #1 l0k1

_{Go to steemit.com/~witnesses to cast your vote by typing l0k1 into the text entry at the bottom of the leaderboard.}

_{(note, my username is spelled El Zero Kay One or Lima Zero Kilo One, all lower case)}