[Steem on RPi] Downloading block files...

I have succeeded to build the steem source for arm and made it a docker image.
Then, I could run a witness node on Raspberry Pi 4B (8G RAM).

Related Posts

• 라즈베리파이로 증인노드를 돌려본다? - Tutorial - 일단 실패
• Raspberry Pi 4B로 증인노드를 돌려본다? - 성공 흔적을 찾아서!
• [Steem on RPi] Other links
• [STEEM on RPi] steemd Built anyway on arm64
• [Steem On RPi] 빌드 성공과 핫도그
• [Steem on RPi] It works! - Screenshots and Logs
  [Steem on RPi] How to Build steem for ARM64

The final question is whether or not the RPi4's hardware can sync and run a witness node.
To do so, I need a latest block file and have been downloading the files here:
https://files.steem.fans/ (thanks to @ety001).

Hi, I have been downloading latest block files to quickly sync on RPi4.

There are two ways to download block files.

1 Stream way

As @ety001 guided, you can download the compressed file and decompress it to a destination which will be data/witness_node_data_dir.

For the disk less space users, you could try the stream way to download and decompress backup data. Like this:

wget -c -qO- https://files.steem.fans/data/steem_witness-20210806.tar.lz4 | lz4 -d | tar x

I tried this but failed because my home internet connection was not stable. The tar command always overrides the existing file. So if the download stops, then you have to download the file from the start, which was not possible for most of case because the file size is too large ~ 200GB.

2 Download first and decompress way

The other option is to first download the file and decompress it in the right directory.

wget -c command allows continuous downloading. This is more reliable way.

steem_witness-20210806.tar.lz4       55%[++++++++++++++++++++++++++++++++===>                             ] 117.71G  2.62MB/s    eta 11h 32m

But my problem is slow network. As you can see the speed is 2.6MS/s. I was much lower yesterday. It should be around 10MB/s but recently the speed became so low. And my USB to Ethernet adapter was out of order recently. The cheap stuff always cause trouble.

After downloading the file, do the following to decompress it in the right folder.

• go to the witness_node_data_dir~~
• unzip the lz4 file and untar the result using the pipe

$ cd ~/steem-docker-ex/data/witness_node_data_dir
$ lz4 -d /media/ssd/steem/steem_witness-20210806.tar.lz4 | tar xv

$ lz4 -d steem_witness-20210806.tar.lz4 | tar xv - -C ~/steem-docker-ex/data/witness_node_data_dir/

Here I am using external ssd and attached it to my RPi4.

There will be the blockchain folder under the witness_node_data_dir.
The file will be decompressed 20MB per second, which mean takes looooong time. Keep watching the progress!

Update

I have trouble in decompressing the file and copy the files at the same time.
I tried two times the decompressing and copying the files but it failed at 160GB.
I did this command on RPi4 (8G RAM). I guess the RPI's RAM is not enough for this task.

Before, I could decompress (without tar) in the witness_node_data_dir, so I thought there would be no problem.
But I guess it is.

If you have large storage not like me, then you will have not issue here.

This time I am trying to decompress and copy the files on i7 machine which has 16GB RAM. I attached both the ssd having the lz4 file and RPi's SD card to the i7 machine in which the decompression and copy occur.

Let's see the result. If this would be not successful, I have to erase lots of data to secure free storage. Wish me luck.

Oh my god, the source file has an error. The decompression stopped at the exact same place: 160GB. oops

Decompressed : 162900 MB  Error 66 : Decompression error : ERROR_decompressionFailed
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

So I tried to keep downloading the source, but it is completed. oops.

 The file is already fully retrieved; nothing to do.

I think I have to try other source.. long way to go.

---

TL; DR

I recommend you separate downloading and decompressing if your network is not stable.

When the downloading and decompressing finish, I could start lauch steem docker to sync block data.

Wish Me Luck, It is just before the new era of witness nodes

Resteem / Vote / Follow / Comment / Support

cc.
@steemcurator01
@steemitblog

Does the foundation have some interest in this project? Want to make it official support? Just let me know.