Part 2: Streaming EOS Blocks With Python

This tutorial is part of a series where different aspects of programming with EOS are explained. This part will look at streaming blocks from the EOS Blockchain by using Python. The full curriculum can be found below.

---

Repository

https://github.com/EOSIO/eos

What will I learn

• Create an iterable block stream
• Increase performance by using a session
• Dealing with status codes

Requirements

• Python3.6

Difficulty

• basic

---

Tutorial

Setup

Download the file from Github. There is 1 file get_blocks.py. Running the file without any arguments will initiate a performance test, it is also possible to run get_blocks.py with 2 arguments start_block and block_count.

Run scripts as following:
> python get_blocks.py
or
> python get_blocks.py 1 100

Create an iterable block stream

The previous tutorial looked at how to retrieve EOS blocks by using API POST requests. Expanding on this code a basic version for streaming these block would be to build a loop. In order to have iterable output yield can be used.

def stream_blocks(start_block, block_count):
    for block_num in range(start_block, start_block + block_count):
            yield get_block(block_num)

This makes the function callable as a list which can be iterated, which in turn simplifies processing the blocks afterwards.

for block in stream_blocks(start_block, block_count):
    # perform action

Increase performance by using a session

The problem with looping standard POST requests is that the performance will suffer. For every request a new connection has to be established. Instead a Session can be created which is then used to handle all the requests.

import requests


s = requests.Session()
request = s.post(url=url, data=parameters)

The session needs to be created outside of the loop.

s = requests.Session()

for block_num in range(start_block, start_block + block_count):
    yield get_block(block_num, s)

Dealing with status codes

The increased performance creates a new set of problems. Depending on which api_endpoint is being used there might be restrictions on how many requests can be made in a specific time frame. Which can be solved by adding a delay time.sleep() before each request.

EOS has a block_time of 500 ms or 2 blocks per second. This means that when the head_block is reached requests can be made for blocks that do not exist yet. Each request comes with a status_code, 200 means that everything is ok. There are several status_codes that can indicate different kind of problems. EOS nodes use the status_code 500 when a block that does not exist is requested.

{
    "code": 500,
    "message": "Internal Service Error",
    "error": {
        "code": 3100002,
        "name": "unknown_block_exception",
        "what": "unknown block",
        "details": [{
            "message": "Could not find block: 6000000",
            "file": "chain_plugin.cpp",
            "line_number": 832,
            "method": "get_block"
        }]
    }
}

By creating a loop that only returns when the correct status_code the stream will halt until the issue has been dealt with.

while True:
    request = s.post(url=url, data=parameters)
    if verify_request(request):
        return request.text

This allows for unique handling of each status_code.

def verify_request(request):
    if request.status_code == 200:
        return True
    elif request.status_code == 500:
        # Deal with the problem

This can be taken a step further by looking at the error codeinside the text part of the request.

    elif request.status_code == 500:
        error_code = json.loads(request.text)['error']['code']
        if error_code == 3100002:
            print('Waiting for blockchain to catch up\n')
            time.sleep(0.5)

Running the code

Running the code without any arguments will initiate the performance test. 20 blocks will be retrieved with and without using a session. The total time for requesting all blocks will be displayed.

python get_blocks.py

Starting performance test without session

Block 1
Block 2
.
.
Block 20

Took 12.307921886444092 seconds for completion

Starting performance test with session

Block 1
Block 2
.
.
Block 20

Took 3.4480578899383545 seconds for completion

In addition running the code with the arguments start_block and block_count will stream the block_count amount of blocks from start_block.

python get_blocks.py 1 10
Block 1
{"timestamp":"2018-06-08T08:08:08.500","producer":"","confirmed":1,"previous":"0000000000000000000000000000000000000000000000000000000000000000","transaction_mroot":"0000000000000000000000000000000000000000000000000000000000000000","action_mroot":"aca376f206b8fc25a6ed44dbdc66547c36c6c33e3a119ffbeaef943642f0e906","schedule_version":0,"new_producers":null,"header_extensions":[],"producer_signature":"SIG_K1_111111111111111111111111111111111111111111111111111111111111111116uk5ne","transactions":[],"block_extensions":[],"id":"00000001405147477ab2f5f51cda427b638191c66d2c59aa392d5c2c98076cb0","block_num":1,"ref_block_prefix":4126519930}
Block 2
{"timestamp":"2018-06-09T11:56:30.000","producer":"eosio","confirmed":0,"previous":"00000001405147477ab2f5f51cda427b638191c66d2c59aa392d5c2c98076cb0","transaction_mroot":"0000000000000000000000000000000000000000000000000000000000000000","action_mroot":"e0244db4c02d68ae64dec160310e247bb04e5cb599afb7c14710fbf3f4576c0e","schedule_version":0,"new_producers":null,"header_extensions":[],"producer_signature":"SIG_K1_KhKRMeFHa59AzBaqNvq89Mye9uTNsRsY4koYZk4GBxb4UfSEakj4LwxxP5xQVK4q9N32JFhMpjnHa8pgTKNLwP1vXpU6eg","transactions":[],"block_extensions":[],"id":"0000000267f3e2284b482f3afc2e724be1d6cbc1804532ec62d4e7af47c30693","block_num":2,"ref_block_prefix":976177227}
.
.
.

Curriculum

Part 1: Using Python To Make API Requests To EOS Public API Endpoints

---

The code for this tutorial can be found on GitHub!

This tutorial was written by @juliank.