Steem Full Node Syncing progress - October 27, 2017steemCreated with Sketch.

in witness •  21 days ago

Hi everyone,

Today, I'd like to show you the progress so far syncing my Full Steem Node:

Shared Memory file reading:

After using the virtual memory of the 32GB of RAM of the machine, Steemd began reading from the SSD. This can be observed in the following image, where you can see that the NVMe drive is being accessed for reading:

1.png

Overall, read and write speeds are really fast, but even with these speeds, Steemd is using between 50% to 70% of a single core of the CPU, which slows down the sync process:

Screenshot of nmon:

2.png

Screenshot of iotop:

3.png

Screenshot of the System Monitor:

5.png

The Syncing progress:

As of the time of this post, the syncing is done up to 78.2361%. Yesterday, I installed Ubuntu natively instead of running it in a virtual machine to see if the performance improves. It definitely is somewhat faster, but then again, my preference of running it in a VM over physical is that I could perform regular snapshots/checkpoints of the machine in case of a failure.

The sync process started yesterday at 5:30 PM approximately:

4.png

Will adding another 32GB of RAM improve the speed? I highly doubt it, because after Steemd used the virtual memory assigned on the 32GB of RAM the machine has, the SSD speed was fast enough to keep it syncing at a fast pace. The slowdown started after some percents after hitting HF18. Right now, I'm at the current HF19 but still a long way to go until it syncs completely.

Yesterday night, at about 9PM, the sync process was done up to 74%. Today at 9AM, it was at the same 78% you're seeing right now, but without the extra lines saying about the Free Memory available.

Between percents, it is using 2GB of Shared Memory as of the 75% mark, and I believe this goes even higher as the sync process continues.

SSD read speeds seems to be fine, as, as seen in the first image from nmon, it seems to be using less I/O. Compared to the writings, the transfers/sec, which can be compared to IOPS, are a lot smaller than when it writes the data to the SSD. It seems that either Linux or steemd are not optimized to take the full potential of the SSD, as it can handle more than 200K IOPS. Those 200K IOPS are seen when linux flushes the buffer to the SSD, but cannot be seen when it is reading. You can see the write performance in yesterday's post by clicking here.

I did installed Linux Kernel 4.13 after installing Ubuntu, as Ubuntu 16.04.3 ships with 4.10. I don't know if this improves the performance but everything is going allrigh up to this point.

How many days more until the sync finishes? Maybe up to a week at this pace, I really don't know, but this is a journey I'm definitely enjoying 🙂

Have a nice day and thanks for reading this!

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!