Zero to Witness: Part 5: Downloading the Blockchain

rexthetech (68)in Witness Activities • 2 years ago (edited)

Finally, we have a well-oiled server and we're ready to download the Steem blockchain.

But first, a little something we really should have done back in Part 3!

Set the hostname

Let's set the server's hostname to witnessnode. Log in as the steem user over SSH, and do:

sudo hostnamectl set-hostname witnessnode

If you log out and in again, your prompt will now show "steem@witnessnode".

Much prettier! And, if you get to the point where you're working with multiple servers, it will save a certain amount of sanity.

Install prerequisites

There are a couple of tools we'll need in order to run Steem.

Docker is a handy way to package up an application into a container which holds everything required to run the application, including dependencies such as software libraries and config files. @ety001 maintains a Steem docker container which is the easiest way to run an up-to-date Steem node.

Screen is a super handy util which ensures a command that we've started on the server will keep running even if our connection breaks. We're about to do a massive blockchain download onto the server, so using Screen for that will prevent heartache if our SSH connection dies for any reason during the download.

Let's use apt to install Docker and Screen. We run apt update before install, to ensure apt knows about the latest versions of everything.

Log into the server over SSH as the steem user, and do this:

sudo apt update
sudo apt install docker.io screen

As part of the Docker installation process, a new "docker" user group will have been created on the server. To be able to use Docker, our steem user needs to be a part of that group, so let's do that:

sudo usermod -aG docker steem

How to use Screen

Usually, if we execute a command which takes a long time to run in a normal SSH session, and for whatever reason our SSH connection gets closed before the command completes, it gets killed in the middle of whatever it was doing.

Using Screen allows the command to keep running even if we close our SSH session and come back later. This is exactly what we'll want when downloading the Steem blockchain, which takes a few hours, so let's learn how to use Screen now.

Go ahead and start a screen session:

screen

You'll see a splash screen; hit Enter to dismiss it, and you're inside your new screen session.

It doesn't look like anything has changed! We're still looking at our "steem@witnessnode" command prompt. However, let me show that we're actually inside this resumable session. Let's get something distinctive on the screen. Get a listing of the files in your home directory with:

ls -la

Now, to detach from your Screen session while leaving it running in the background, do Ctrl-A then Ctrl-D.

You'll see that your file listing has disappeared; we're back outside our Screen session, but it's still running.

Just to prove the point that even if your ISP dies (or you just closed your SSH window to have a coffee break), you would still be able to get back to your session: Log out of SSH with exit, and log back in again.

Now reattach to your running session, with:

screen -r

Look at that -- your directory listing is back!

To close the screen session, just type exit from within it. Please exit your current screen session now, just so we're all starting from a known point for the next stage.

Screen summary:

Start a screen session if none is running: screen
Check if there's a screen session running: screen -list
Reconnect to a running session: screen -r
Disconnect from a running session: Ctrl-A, Ctrl-D
Close a running session: exit

Options for syncing the blockchain

There are two main components we need in order to get our Witness Node running: The Steem node software itself (as a Docker container), and the blockchain data.

The blockchain data is the big one, and we've a few ways to get it.

Option 1: We could start the Steem node up, let it connect to the peer-to-peer network to find itself some Seed Nodes, and it will request chunks of blocks from its peers until it's up to date.

This is very slow because the blocks must be parsed individually for our node to build its own block log (a total record of all blocks), and ultimately needs a lot more bandwidth than the next option.

Option 2: We could download an almost-complete copy of the block log as an archive (kindly provided and kept up to date by @ety001 on his site).

This would give our node a massive headstart; it would only need to request the last couple of weeks worth of blocks from its peers (or however long since ety001 last created a block log archive). However, when running as a Witness Node, our Steem node needs to build additional databases from the block log to accelerate certain lookup operations when witnessing blocks.

Although we'd already have the block log, we would need to have our node replay the block log -- step through it block by block -- creating these important local databases.

Option 3: We download an archive containing not only the block log, but the required databases too!

We'll still need to get the most recent couple of weeks of blocks from the peer-to-peer network, but this is certainly the best way to go :-)

The archive download is currently about 230Gb, and decompresses to about 350Gb. There'll be plenty of space on the 1Tb drive that I'm using in the AX51.

Oh, one important note which may be obvious but still bears mentioning. At no point are you downloading this massive archive to your own machine. We're instructing the server to download it, and the only traffic to your own internet connection is the text you see in your SSH session. You don't need to worry about your domestic bandwidth allocation :)

Downloading the blockchain

Having made sure you don't have a screen session running, start one:

screen

Now, go to @ety001's Steem Data Server, and look for the file near the top that starts with steem_witness. Right-click on it in your browser to copy the link. At time of writing, it's https://files.steem.fans/s3/steem_witness_20220805.tar.lz4.

Let's start the download, using the command line downloading tool wget, which is preinstalled with Ubuntu. Substitute in the address that you copied above:

wget https://files.steem.fans/s3/steem_witness_20220805.tar.lz4

You'll see that wget gives us a progress bar and time estimation, and that the download will take a few hours. It's safely running inside Screen, so it doesn't matter if we close the SSH session.

Overview

Well, I've proofread this massive article a few times, and the download's still in very early stages here, so this seems like a good point to end for today and go and enjoy the sunshine :-)

Next time, we'll unpack the blockchain archive and get the config file ready for our Steem node.