Part 3: Unending Block Stream, Head_block / Last_irreversible_Block and Multiple Api_endpoint Support

This tutorial is part of a series where different aspects of programming with EOS are explained, the full curriculum can be found below. This part will focus on building an unending stream with fallback api_endpoints and goes into more depth about the head_block and irreversible blocks.

---

Repository

https://github.com/EOSIO/eos

What will I learn

• Head block & last irreversible block
• Unending stream
• Multiple api_endpoints and switching

Requirements

• Python3.6

Difficulty

• basic

---

Tutorial

Setup

Download the file from Github. There is 1 file stream_blocks.py. Running the file without any arguments will test the api_endpoints for their speed to decide what node is best and then start streaming from the current head_block. Adding the argument irreversible will perform the same test but only stream irreversible blocks.

Run scripts as following:
> python stream_blocks.py

Head block & last irreversible block

On the EOS Blockchain new blocks are produced by 21 block producers (BPs) every halve a second. As this is not enough time for the blocks to propagate over the entire network BPs are chosen to produce a set amount of blocks before another BP takes the role as a producer. The frequency in which the BPs are chosen is based on the amount of votes this BP has received.

This means that the current block producing BP is slightly ahead of the other BPs. For this reason there is a distinction between the head_block and the last_irreversible_block. Every block that has been verified by allBPs becomes an irreversible block. As it takes time for the blocks to propagate over the network and get verified by all BPs there is a delay before a block becomes irreversible.

eosnetworkmonitor

Depending of the type of application use of the head_block or last_irreversible_block can be preferred. For example dealing with financial transaction using the last_irreversible_block is recommended, while building a chat application speed is more important so the current head_block is preferred. Information about the head_block and last_irreversible_block is found by using the get_info API request.

def get_head_block(self, irreversible=None):
    api_call = '/v1/chain/get_info'
    address = self.url + api_call

    request_json = self.s.get(url=address).json()

    if not irreversible:
        return request_json['head_block_num']
    else:
        return request_json['last_irreversible_block_num']

Unending stream

Knowing this a robust unending stream can be made. First the start_block has to be decided. If none is given it is taken from the current head_block or last_irreversible_block which is set in self.block. This is to keep track of which blocks have been processed in case the stream crashes and needs to restart, starting at self.block.

self.block = self.get_head_block(irreversible)

def stream_blocks(self,
                  start_block=None,
                  block_count=None,
                  irreversible=None):

    if not start_block:
        start_block = self.block
    else:
        self.block = start_block

When upgrading code for the application where it has to be shut down it can be preferred to start at the last processed block. In which case a start_block can be set.


The next block of codes takes the start_block and then ask for the current head_block / last_irreversible_block to create a range ofblocks. These blocks are requested and after the start_block is set to the last requested block. Repeating indefinitely.

The upside of this is that it becomes impossible to request blocks ahead of the current head_block / last_irreversible_block block. The downside is that it requires additional API requests to retrieve the information about the head_block / last_irreversible_block. In the case of using the last_irreversible_block this is the only way to make sure the blocks being requested are irreversible. As it is possible to requests blocks ahead of the last_irreversible_block, so no error message would be returned if that were the case.

while True:
        head_block = self.get_head_block(irreversible)

        for n in range(start_block, head_block + 1):
            yield self.get_block(n)
            self.block += 1

        start_block = head_block + 1

In case something unexpected happens, such as losing internet connection the stream is put into a while loop with a try and except clause. This will keep trying to restart the stream.

while True:
    try:
        for block in self.stream_blocks(irreversible):
            print(f'\nBlock {self.block}')
    except Exception as error:
            time.sleep(60)
            print(repr(error))

Multiple api_endpoints and switching

Problems can also occur at the api_endpoint side. Proving additional api_endpoints to fall back to solves this problem.

self.nodes = ['http://mainnet.eoscanada.com',
              'http://api-mainnet1.starteos.io',
              'http://api.eosnewyork.io']
self.node_index = 0
self.url = self.nodes[self.node_index]

In the case the api_endpoint returns an unexpected error the node_index will be increased and adjusted to stay in the limits of the nodes list. The next api_endpoint is then selected and the session is renewed.

def reset_api_endpoint(self):
    self.node_index = (self.node_index + 1) % len(self.nodes)
    self.url = self.nodes[self.node_index]
    self.s = requests.Session()
    print(f'New api_endpoint: {self.url}')

Additionally when starting the stream all the nodes are tested for their response time. For each api_endpoint 100 blocks are retrieved and it is timed how long this process takes. The results are added to the results list as tuples (url, time). Then the results are ordered by their time sort(key=operator.itemgetter(1)). Eventually a sorted nodes list is returned by stripping only the urls from the each tuple in results.

def test_api_endpoints(self):
    results = []

    for url in self.nodes:
        self.url = url
        start_time = time.time()
        print(f'\nTesting: {url}')

        for block in self.stream_blocks(start_block=1, block_count=100):
            print(f'{self.block}/100', end='\r')

        end_time = time.time() - start_time
        results.append((url, end_time))
        print(f'Took {end_time:.4} seconds for completion')

    results.sort(key=operator.itemgetter(1))
    self.nodes = [node[0] for node in results]
    print(self.nodes)

Running the code

The code can be run in two modes, either the current head_block mode or the last_irreversible_block mode. Regardless of the mode a test of each api_endpoint will first be conducted to sort the api_endpoints by their speed.

python stream_blocks.py irreversible

Testing: http://mainnet.eoscanada.com
Took 27.98 seconds for completion

Testing: http://api-mainnet1.starteos.io
Took 33.16 seconds for completion

Testing: http://api.eosnewyork.io
Took 29.68 seconds for completion
['http://mainnet.eoscanada.com', 'http://api.eosnewyork.io', 'http://api-mainnet1.starteos.io']

Mode: last_irreversible_block
Block 5998244

Curriculum

• Part 1: Using Python To Make API Requests To EOS Public API Endpoints
• Part 2: Streaming EOS Blocks With Python

---

The code for this tutorial can be found on GitHub!

This tutorial was written by @juliank.