Introduction

Blockchains often stores state, consensus information and other meta data in a single file. Recently there are mechanisms developed to store state data off chain. But the need to read, process and reindex "blockchain" is a major challenge in starting new nodes as it consumes hours. To add to it, most of the blockchains doesn't use multi threading making it even more slower.

To avoid I/O bottle necks blockchains depends on shared memory and thus large amounts of RAM. Conventionally the Proof of Stake (PoS) algorithms thus have high Space Complexity aka memory requirements. Graphene and DPoS consensus based blockchains like STEEM also addresses the I/O issue by making use of large amounts of memory. In the recent past STEEM has reduced the memory requirements by implementing efficient snapshot mechanisms. Other chains with similar challenges will Bitshares, SCORUM and may be EOS.

In this scenario, I got a chance to evaluate the performance of Intel's new Octane NVMe Flash technology to replace RAM.

Problem Statement

As briefly mentioned in the introduction, the requirement for large amounts of RAM is not just a cost barrier but in some cases like running full RPC nodes the availability of hardware makes it difficult to get started. The recent developments in flash technology from Intel and the introduction of Octane drives gives us a cheaper, more accessible option to run memory intensive workloads.

In this specific case, I wanted to test whether its possible to run a STEEM full node using Optane NVMe instead of RAM. I was lucky enough to get hands on a reasonable hardware with Intel 900P Octane drive on it.

Testing

My first attempt to run a full node failed to unknown reasons. I got help from @jesta on CPU types& general guidelines to set things up. The test is not a very scientific one and I had some additional services like SNMP running on it. To make matters worse, I have a HDD on which the OS is installed.

Environment

A dedicated server with 1 G/s network connectivity and minimal firewall settings. Default system services that comes with Ubuntu 16.04 LTS left untouched.

---

OS: Ubuntu 16.04

Kernel Version: 4.15.0-38-generic

CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz

Optane Drive : INTEL SSDPED1D480GA ( Intel Optane 900P 480GB )

---

Number of Open files in the system at the start of the test:

[root@STEEM-Full-NODE:/chain]# lsof | wc -l
3054
[root@STEEM-Full-NODE:/chain]#

---

The test was done with minimal seed node configuration with only witness and rc_plugin enabled. The blockchain file was kept under /chain where the Optane drive is mounted. The shared memory file also was kept under /chain.

Minimal tests and results

@drakos was kind enough to guide me and provide me with few handy (new ) commands and useful options to others. Appreciate the help :)

At one point I was doubtful whether this is really an Optane and @drakos suggested lsblk which is new to me.

[root@STEEM-Full-NODE:/chain/0]# lsblk -io KNAME,TYPE,SIZE,MODEL
KNAME TYPE SIZE MODEL
sda disk X.XT ATXA8123NM0033
sda1 part 953M
sda2 part X.XT
sda3 part 3.8G
nvme0n1 disk 447.1G INTEL SSDPED1D480GA

---

hdparm with the direct option

From the man(ual) page:

             Use the kernel´s "O_DIRECT" flag when performing a -t timing test.  This bypasses the page cache, causing the reads to go directly from the drive into hdparm's buffers, using so-called "raw" I/O.  In many cases, this can produce results that appear much faster than the usual page cache method, giving a better indication of raw  device  and  drive performance.

---

hdparm -tT --direct /dev/nvme0n1

[root@STEEM-Full-NODE:~]#sync ; echo 3 > /proc/sys/vm/drop_caches ; lsof | grep /chain ;
[root@STEEM-Full-NODE:~]#hdparm -tT --direct /dev/nvme0n1

/dev/nvme0n1:
Timing O_DIRECT cached reads: 4848 MB in 2.00 seconds = 2425.03 MB/sec
Timing O_DIRECT disk reads: 7258 MB in 3.00 seconds = 2418.92 MB/sec
[root@STEEM-Full-NODE:~]#:

---

[root@STEEM-Full-NODE:~]#hdparm -tT /dev/nvme0n1

/dev/nvme0n1:
Timing cached reads: 19652 MB in 1.99 seconds = 9855.17 MB/sec
Timing buffered disk reads: 7680 MB in 3.00 seconds = 2559.88 MB/sec

These are advertised values.

---

The memory values looked something like this little while after the replay.

free -h

I wanted to tweak swapiness and cache pressure but the sysctl parameters were kept at the values suggested by STEEM build instructions.

[root@STEEM-Full-NODE:/chain/0]# optimize() { echo 75 | sudo tee /proc/sys/vm/dirty_background_ratio; echo 1000 | sudo tee /proc/sys/vm/dirty_expire_centisecs; echo 80 | sudo tee /proc/sys/vm/dirty_ratio; echo 30000 | sudo tee /proc/sys/vm/dirty_writeback_centisecs; }
[root@STEEM-Full-NODE:/chain/0]#optimize

Steemd build options and disk layout

Steem was buld with low memory options

cmake -DLOW_MEMORY_NODE=ON -DCLEAR_VOTES=ON -DSTEEM_STATIC_BUILD=ON -DSKIP_BY_TX_ID=O -DCMAKE_BUILD_TYPE=Release SOURCE

Everything related to steem in /chain

/dev/nvme0n1 440G 201G 216G 49% /chain

Summary

The replay completed in less than 7 hours & is considerably faster than SSD.

Load conditions are normal and no swap was used.

This write up can be considered as a starting point/tutorial for anyone who wants play with Optane drives. Additional testing is needed to make sure that this setup works in witness mode. I hope to get access to devices with the right hardware and perhaps a faster CPU to conduct further tests.

Keywords : Setting up STEEM seed on Intel Optane, Intel Optane 900P, Optane and blockchain