Part 23: Retrieve And Process Full Blocks From The Steem Blockchain

This tutorial is part of a series where different aspects of programming with steem-python are explained. Links to the other tutorials can be found in the curriculum section below. Every action performed on the Steem Blockchain is registered in blocks. These blocks are the building stones for the Steem Blockchain and this tutorial will explain how to retrieve and process individual blocks.

---

Repository

https://github.com/steemit/steem-python

What will I learn

• Stream full blocks from the blockchain
• What do blocks look like?
• What in a transaction?
• What is an operation?
• Filter for different kind of operations

Requirements

• Python3.6
• steem-python

Difficulty

• basic

---

Tutorial

Setup

Download the file from Github. There is 1 file get_block.py which contains the code. The file takes 3 arguments from the command line which sets the starting_block, block_count and what operation to filter for.

Run scripts as following:
> python get_block.py 22224731 10 transfer

Stream full blocks from the blockchain

Tutorials until now used the function stream to stream operations as they appeared on the Steem blockchain from the current head block. This is pretty close to a live stream as long as there is no backlog. This works perfect however it is susceptible to crashes and it is also not possible to backtrack previous blocks.

By taking full control over which blocks to retrieve from the blockchain the user will have more control and the software becomes more resistant to crashes. The Blockchain class holds the function stream_from which can be used to achieve this. The start_block must be set to the preferred block to starting streaming from, block_count is used to calculate the end_block and full_blocks must be set to True in order to retrieve the entire block.

from steem.blockchain import Blockchain

stream = self.b.stream_from(start_block=self.block,
                            end_block=self.end_block,
                            full_blocks=True)
            for block in stream:
              //do something with the block

Note: For this tutorial end_block is set for testing purposes. Leaving this blank will keep the stream alive and wait for new blocks

What do blocks look like?

Blocks are the building stones for the blockchain and in essence contain all the information that is stored on Steem. Blocks are linked in a chain hence the name blockchain. Each block itself is not that large and new blocks are created every 3 seconds. Only information that is added or changed gets put into new blocks. The basic structure of a block looks as follows:

{
    'previous': '01531f5aded177b75bbe144f44958c01e8061890',
    'timestamp': '2018-05-07T15:20:03',
    'witness': 'jesta',
    'transaction_merkle_root': '01fbed04e90d4b19569788f2f34467d2de2edb40',
    'extensions': [],
    'witness_signature': '204cb7d24d1b3b440768873d3bd5fd29dd35925e38508346e998a486a83164b8aa7be6d748fdc4433231275b2caad3fe9400b850ac8656dc4565409115563d9a8b',
    'transactions': []
    'block_id': '01531f5ba3f99e0e10a6015491fa6f09bb3501ac',
    'signing_key': 'STM8MFSHJA2DsjmEUBsf8JYdhm25YghdEUeDEMqNVMrWMCgHErDaL',
    'transaction_ids': []
}

The timestamp is useful to validate the age of the data, transactions and transaction_ids are left blank in this example as they contain most of the data. The transaction and the transaction_id are associated with each other based on their index. So the first transaction correlates with the first transaction_id. Every action on the Steem network registered into a transaction.

What is a transaction?

Every transaction has the following structure:

{
    'ref_block_num': 8004,
    'ref_block_prefix': 1744181707,
    'expiration': '2018-05-07T15:29:54',
    'operations': [
        ['vote', {
            'voter': 'acehjaya',
            'author': 'zunaofficial79',
            'permlink': 'kabut-pagi',
            'weight': 10000
        }]
    ],
    'extensions': [],
    'signatures': ['1f205086670c27365738697f0bc7cfde8b6e976608dc12b0d391b2b85ad7870a002313be4b09a358c30010be2a09bebacead866ba180afc348ce40c70566f4ed88']
}

Found under operations are alterations made to the Steem Blockchain by its users. Here common operations like voting, transfers and comments are found.

What is an operation

Every action that is registered on Steem is a operation which fits in one of the following categories:

comment, delete_comment, vote
account_create, account_update, request_account_recovery, recover_account
limit_order_create, limit_order_cancel
transfer, transfer_to_vesting, withdraw_vesting, convert
pow, feed_publish, witness_update, account_witness_vote
custom, custom_json

Source


Each operation has its own structure and requires unique handling for processing. A transfer looks as follows:

{
    'from': 'yusril-steem',
    'to': 'postpromoter',
    'amount': '5.000 SBD',
    'memo': 'https://steemit.com/life/@yusril-steem/our-challenges-and-trials-in-life'
}

Filter for different kind of operations

Each block is filled with transactions in which an operation is embedded. Filtering for operations is done by checking its type which is located under ['operations'][0][1]. The transaction_index is tracked to know the specific transaction in which the operation took place.

for transaction in block['transactions']:
    if transaction['operations'][0][0] == self.operation:
        print(transaction_index,
              transaction['operations'][0][1])
    transaction_index += 1

Running the script

Running the script will print operations, for which the filter is set, to the terminal ,with their corresponding transaction_index, starting from the starting_block for block_count amount of blocks. Feel free to test out the different operation types to get a feeling how they are stored in the blocks.

python get_block.py 22224731 10 transfer

Booted
Connected to: https://rpc.buildteam.io

Block:  22224731
44 {'from': 'hottopic', 'to': 'jgullinese', 'amount': '0.001 SBD', 'memo': 'Hello Friend, Your post will be more popular and you will find new friends.We provide "Resteem upvote and promo" service.Resteem to 18.000+ Follower,Min 45+ Upvote.Send 1 SBD or 1 STEEM to @hottopic (URL as memo) Service Active'}
...
...
...

Curriculum

Set up:

• Part 0: How To Install Steem-python, The Official Steem Library For Python
• Part 1: How To Configure The Steempy CLI Wallet And Upvote An Article With Steem-Python

Filtering

• Part 2: How To Stream And Filter The Blockchain Using Steem-Python
• Part 6: How To Automatically Reply To Mentions Using Steem-Python

Voting

• Part 3: Creating A Dynamic Autovoter That Runs 24/7
• Part 4: How To Follow A Voting Trail Using Steem-Python
• Part 8: How To Create Your Own Upvote Bot Using Steem-Python

Posting

• Part 5: Post An Article Directly To The Steem Blockchain And Automatically Buy Upvotes From Upvote Bots
• Part 7: How To Schedule Posts And Manually Upvote Posts For A Variable Voting Weight With Steem-Python

Constructing

• Part 10: Use Urls To Retrieve Post Data And Construct A Dynamic Post With Steem-Python

Rewards

• Part 9: How To Calculate A Post's Total Rewards Using Steem-Python
• Part 12: How To Estimate Curation Rewards Using Steem-Python
• Part 14: How To Estimate All Rewards In Last N Days Using Steem-Python

Transfers

• Part 11: How To Build A List Of Transfers And Broadcast These In One Transaction With Steem-Python
• Part 13: Upvote Posts In Batches Based On Current Voting Power With Steem-Python

Account Analysis

• Part 15: How To Check If An Account Is Following Back And Retrieve Mutual Followers/Following Between Two Accounts
• Part 16: How To Analyse A User's Vote History In A Specific Time Period Using Steem-Python
• Part 18: How To Analyse An Account's Resteemers Using Steem-Python

Wallet

• Part 22: Calculate Account Value Based On Current Market Prices Via API

---

The code for this tutorial can be found on GitHub!

This tutorial was written by @juliank.