Part 2: Streaming EOS Blocks With Python

in utopian-io •  4 months ago

steem-python.png

This tutorial is part of a series where different aspects of programming with EOS are explained. This part will look at streaming blocks from the EOS Blockchain by using Python. The full curriculum can be found below.


Repository

https://github.com/EOSIO/eos

What will I learn

  • Create an iterable block stream
  • Increase performance by using a session
  • Dealing with status codes

Requirements

  • Python3.6

Difficulty

  • basic

Tutorial

Setup

Download the file from Github. There is 1 file get_blocks.py. Running the file without any arguments will initiate a performance test, it is also possible to run get_blocks.py with 2 arguments start_block and block_count.

Run scripts as following:
> python get_blocks.py
or
> python get_blocks.py 1 100

Create an iterable block stream

The previous tutorial looked at how to retrieve EOS blocks by using API POST requests. Expanding on this code a basic version for streaming these block would be to build a loop. In order to have iterable output yield can be used.

def stream_blocks(start_block, block_count):
    for block_num in range(start_block, start_block + block_count):
            yield get_block(block_num)



This makes the function callable as a list which can be iterated, which in turn simplifies processing the blocks afterwards.

for block in stream_blocks(start_block, block_count):
    # perform action


Increase performance by using a session

The problem with looping standard POST requests is that the performance will suffer. For every request a new connection has to be established. Instead a Session can be created which is then used to handle all the requests.

import requests


s = requests.Session()
request = s.post(url=url, data=parameters)



The session needs to be created outside of the loop.

s = requests.Session()

for block_num in range(start_block, start_block + block_count):
    yield get_block(block_num, s)

Dealing with status codes

The increased performance creates a new set of problems. Depending on which api_endpoint is being used there might be restrictions on how many requests can be made in a specific time frame. Which can be solved by adding a delay time.sleep() before each request.

EOS has a block_time of 500 ms or 2 blocks per second. This means that when the head_block is reached requests can be made for blocks that do not exist yet. Each request comes with a status_code, 200 means that everything is ok. There are several status_codes that can indicate different kind of problems. EOS nodes use the status_code 500 when a block that does not exist is requested.

{
    "code": 500,
    "message": "Internal Service Error",
    "error": {
        "code": 3100002,
        "name": "unknown_block_exception",
        "what": "unknown block",
        "details": [{
            "message": "Could not find block: 6000000",
            "file": "chain_plugin.cpp",
            "line_number": 832,
            "method": "get_block"
        }]
    }
}



By creating a loop that only returns when the correct status_code the stream will halt until the issue has been dealt with.

while True:
    request = s.post(url=url, data=parameters)
    if verify_request(request):
        return request.text



This allows for unique handling of each status_code.

def verify_request(request):
    if request.status_code == 200:
        return True
    elif request.status_code == 500:
        # Deal with the problem



This can be taken a step further by looking at the error codeinside the text part of the request.

    elif request.status_code == 500:
        error_code = json.loads(request.text)['error']['code']
        if error_code == 3100002:
            print('Waiting for blockchain to catch up\n')
            time.sleep(0.5)

Running the code

Running the code without any arguments will initiate the performance test. 20 blocks will be retrieved with and without using a session. The total time for requesting all blocks will be displayed.

python get_blocks.py

Starting performance test without session

Block 1
Block 2
.
.
Block 20

Took 12.307921886444092 seconds for completion

Starting performance test with session

Block 1
Block 2
.
.
Block 20

Took 3.4480578899383545 seconds for completion



In addition running the code with the arguments start_block and block_count will stream the block_count amount of blocks from start_block.

python get_blocks.py 1 10
Block 1
{"timestamp":"2018-06-08T08:08:08.500","producer":"","confirmed":1,"previous":"0000000000000000000000000000000000000000000000000000000000000000","transaction_mroot":"0000000000000000000000000000000000000000000000000000000000000000","action_mroot":"aca376f206b8fc25a6ed44dbdc66547c36c6c33e3a119ffbeaef943642f0e906","schedule_version":0,"new_producers":null,"header_extensions":[],"producer_signature":"SIG_K1_111111111111111111111111111111111111111111111111111111111111111116uk5ne","transactions":[],"block_extensions":[],"id":"00000001405147477ab2f5f51cda427b638191c66d2c59aa392d5c2c98076cb0","block_num":1,"ref_block_prefix":4126519930}
Block 2
{"timestamp":"2018-06-09T11:56:30.000","producer":"eosio","confirmed":0,"previous":"00000001405147477ab2f5f51cda427b638191c66d2c59aa392d5c2c98076cb0","transaction_mroot":"0000000000000000000000000000000000000000000000000000000000000000","action_mroot":"e0244db4c02d68ae64dec160310e247bb04e5cb599afb7c14710fbf3f4576c0e","schedule_version":0,"new_producers":null,"header_extensions":[],"producer_signature":"SIG_K1_KhKRMeFHa59AzBaqNvq89Mye9uTNsRsY4koYZk4GBxb4UfSEakj4LwxxP5xQVK4q9N32JFhMpjnHa8pgTKNLwP1vXpU6eg","transactions":[],"block_extensions":[],"id":"0000000267f3e2284b482f3afc2e724be1d6cbc1804532ec62d4e7af47c30693","block_num":2,"ref_block_prefix":976177227}
.
.
.

Curriculum

Part 1: Using Python To Make API Requests To EOS Public API Endpoints


The code for this tutorial can be found on GitHub!

This tutorial was written by @juliank.

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Thank you for your contribution @steempytutorials.

I liked your tutorial, but if the tutorial was more detailed I liked it more!

Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.

To view those questions and the relevant answers related to your post, click here.


Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]

Hey @steempytutorials
Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!

Want to chat? Join us on Discord https://discord.gg/h52nFrV.

Vote for Utopian Witness!