Store the Blockchain in a flat file with Python

For anyone doing work with Steem data, having a flat file of the entire Blockchain is very convenient for fast, bulk access.

The only dependency is the Official Python Library, which we can install with:

pip install -U steem

The script

import json
import os
from contextlib import suppress
from steem.blockchain import Blockchain


def get_last_line(filename):
    if os.path.isfile(filename):
        with open(filename, 'rb') as f:
            f.seek(-2, 2)
            while f.read(1) != b"\n":
                f.seek(-2, 1)
            return f.readline()


def get_previous_block_num(block):
    if not block:
        return -1

    if type(block) == bytes:
        block = block.decode('utf-8')

    if type(block) == str:
        block = json.loads(block)

    return int(block['previous'][:8], base=16)


def run(filename):
    b = Blockchain()
    # automatically resume from where we left off
    start_block = get_previous_block_num(get_last_line(filename)) + 2  # previous + last + 1
    with open(filename, 'a+') as file:
        for block in b.stream_from(start_block=start_block, full_blocks=True):
            file.write(json.dumps(block, sort_keys=True) + '\n')


if __name__ == '__main__':
    output_file = '/home/user/Downloads/steem.blockchain.json'
    with suppress(KeyboardInterrupt):
        run(output_file)

The only thing that needs changing is this line, which indicates a location where the flat file will be stored:

 output_file = '/home/user/Downloads/steem.blockchain.json'

You can run this script as many times as you like, and it will continue from the last block it synced.

Useful Utilities

To see how many blocks we currently have, we can simply perform a line count.

wc -l steem.blockchain.json

To inspect a particular block, and pretty-print it:

sed '10000q;d' steem.blockchain.json | python -m json.tool

Replace 10000 with desired block_number + 1.