Developers Guide to Steem's Blockchain

in #steem8 years ago

I have been building a blockchain parser for my Steem Fundamental Analysis project, and I thought it would be a good idea to document my findings.

If you're a developer and you'd like to build an app on top of Steem/Steemit, this guide is for you.

Setting up your node

First, we need to setup a node. There are plenty of guides and tools (for Docker, Ubuntu, Windows) on steemit on how to do that, so I won't repeat myself here.

To get access to all the API's, modify your config.ini to have the following values:

enable-plugin = witness account_history tags follow market_history
public-api = database_api login_api network_broadcast_api follow_api market_history_api

Also, you should remove miner and witness fields. We aren't interested in mining, just running a node.

Now you can run steemd:

./steemd --rpc-endpoint

The --rpc-endpoint will setup a websocket endpoint at ws://

You should be able to connect to it from your favorite programming language (Python, JavaScript, etc.), or by using a tool such as wssh.

If you're using Python, then I suggest the awesome python-steemlib by @xeroc.

You can connect to the RPC like so:

from steemapi.steemnoderpc import SteemNodeRPC

rpc = SteemNodeRPC("ws://", "", "")

Analyzing the Blockchain

Now that we have a node running, and we are connected to it, we can issue commands.

In this tutorial, we will be using the following RPC commands:


Again, the RPC is available for us at ws://localhost:9080.

The full list of commands is available here (scroll to the bottom):

In order to get a block from the blockchain, we want to call a RPC command called get_block, which takes just one argument: Block ID.

Block ID is just a number, anywhere between 0 and the current block ID.
Each new block has an ID at an increment of 1.

You can find out what the current block ID is by looking at steemd output:

651068ms th_a       application.cpp:439           handle_block         ] Got 1 transactions from network on block 3695832

Or by asking steemd via RPC:

 props = rpc.get_dynamic_global_properties()
 last_confirmed_block = props['last_irreversible_block_num']

So to traverse the entire blockchain, from start to finish, we would do something like this:

props = rpc.get_dynamic_global_properties()
current_block = 0
last_confirmed_block = props['last_irreversible_block_num']

while current_block < last_confirmed_block:
    current_block += 1
    print("Processing block %d" % current_block)

    block = rpc.get_block(current_block)

Now lets look at what get_block gives us. If we call rpc.get_block(3332318), we get a block 3332318 in a json format, that looks like this:

{'extensions': [],
 'previous': '0032d8dd7fef5c0eafd7667805134effb9ef21e9',
 'timestamp': '2016-07-19T13:29:15',
 'transaction_merkle_root': '5be8c3f95a29253153e911fd079528ad8216958a',
 'transactions': [{'expiration': '2016-07-19T13:29:24',
                   'extensions': [],
                   'operations': [['vote',
                                   {'author': 'beowulfoflegend',
                                    'permlink': 'how-steemit-self-polices-content',
                                    'voter': 'jills',
                                    'weight': 10000}]],
                   'ref_block_num': 55512,
                   'ref_block_prefix': 2485993294,
                   'signatures': ['20196c89f5e2ca58da3377a30ee08782df59dbd4b72db40ec3348f3417ff653fa84cf684748853e8ea43363c747c74b4ff015626ef9e2b2e902e0ee98a80195336']},
                  {'expiration': '2016-07-19T13:30:10',
                   'extensions': [],
                   'operations': [['comment',
                                   {'author': 'weenis',
                                    'body': 'This is Amazing! Upvoted! \n',
                                    'json_metadata': '',
                                    'parent_author': 'begstreets',
                                    'parent_permlink': 'abstractions-no-1',
                                    'permlink': 'abstractions-no-1',
                                    'title': ''}]],
                   'ref_block_num': 55516,
                   'ref_block_prefix': 765250389,
                   'signatures': ['1f7711cfc441bbc5e6df16d7b73ea7d4ea7d7a40a3c3a1fd857d67b50b30c139d06f742e5fd15a68f2109469c977a12c50abab81632b8a4ba5e328c5048290153b']},
 'witness': 'riverhead',
 'witness_signature': '1f4236f6a0ff79ab2ff9a190162aeb25b5e1509ca44df8803c8091a2abcfe2022f733eb77c15ec8265a1b1d0d2226038b3259db9ebe17b871bcbf3be0dee605bc9'}

We are mostly interested in transactions and operations. Transactions contains a list of operations, which are basically all the events that happened on the blockchain within the 3 second period included in the block.

Operation Types

Currently, the Steem blockchain contains the following operations:

  • comment, delete_comment, vote
  • account_create, account_update, request_account_recovery, recover_account
  • limit_order_create, limit_order_cancel
  • transfer, transfer_to_vesting, withdraw_vesting, convert
  • pow, feed_publish, witness_update, account_witness_vote
  • custom, custom_json

Blog Posts, Comments and Votes

This is a blog post:

=====> comment
{'author': 'rozu15',
 'body': 'content omited becuse it breaks my post',
 'json_metadata': '{"tags":["trump"],"links":[""]}',
 'parent_author': '',
 'parent_permlink': 'trump',
 'permlink': 'us-election-melania-trump-plagiarised-michelle-obama',
 'title': "US election: Melania Trump 'plagiarised' Michelle Obama"}

This is a comment. It has empty title:

=====> comment
{'author': 'densmirnov',
'body': 'nice nickname',
'json_metadata': '{"tags":["steem"]}',
'parent_author': 'steemiscrap',
'parent_permlink': 're-cryptorune-steem-dollar-poloniex-pump-is-a-arbitrage-goldmine-20160719t133438897z',
'permlink': 're-steemiscrap-re-cryptorune-steem-dollar-poloniex-pump-is-a-arbitrage-goldmine-20160719t133602671z',
'title': ''}

The comment has been deleted:

=====> delete_comment
{'author': 'densmirnov',
'permlink': 're-cryptorune-re-densmirnov-re-cryptorune-steem-dollar-poloniex-pump-is-a-arbitrage-goldmine-20160719t133207807z'}

The comment has received a vote:

=====> vote
{'author': 'kain-jc',
 'permlink': 'a-meditation-on-love',
 'voter': 'cantinhodatete',
 'weight': 10000}

If we divide the weight by 100, we get the % of the vote. For instance, a full upvote is 100%. The full downvote is -100%. Half the upvote is 50%.

Account Features

Account has been created:

=====> account_create
{'active': {'account_auths': [],
            'key_auths': [['STM7huHJRrqHE6FDNHJEsUYNNKuQKBfr4iS3xwgyGzXDurbiCRx2V',
            'weight_threshold': 1},
 'creator': 'steem',
 'fee': '5.000 STEEM',
 'json_metadata': '',
 'memo_key': 'STM7R3iScNyQCytvB8EcRdEYLfJCGp3UkyNm7Mt4c9BE1ZoUTdazw',
 'new_account_name': 'saripdol',
 'owner': {'account_auths': [],
           'key_auths': [['STM8XzPFfu3yCujJiSbgdJvKm69LMcm1ckocuvoSaaWRmPRzfH7jr',
           'weight_threshold': 1},
 'posting': {'account_auths': [],
             'key_auths': [['STM8KKqZufL2PM7V9dHeSW8KpN4wWs8cGaiQsz7FCrPohX1ZkfPvv',
             'weight_threshold': 1}}

Account has been updated:

=====> account_update
{'account': 'signaltonoise',
 'active': {'account_auths': [],
            'key_auths': [['STM83ihutAefZSHCJRXLHiRnji8VhQHcouMJc6xGtaLW6PCARXiDa',
            'weight_threshold': 1},
 'json_metadata': '',
 'memo_key': 'STM81hFsfm2csFyPYDCdwjuGJX33EB2Dx6rG74AfFgStu9C6PvzMd',
 'owner': {'account_auths': [],
           'key_auths': [['STM8fcgjrpKbVLHyy3RKnQGx3dhYEdY7bKhbv3FaML5ooK8PpdJaR',
           'weight_threshold': 1},
 'posting': {'account_auths': [],
             'key_auths': [['STM75QK5e69SAuMzVtJqGex3HcaqrPcxpXgxb72WxbUJn2apRX7KX',
             'weight_threshold': 1}}

Transfers, Vesting, Conversions

A transfer of SBD or STEEM from one account to another.

=====> transfer
{'amount': '34.428 SBD',
 'from': 'thisischris225',
 'memo': 'ee1d91d4036200c7',
 'to': 'poloniex'}

Someone has powered-up:

=====> transfer_to_vesting
{'amount': '11.789 STEEM', 'from': 'leksimus', 'to': 'leksimus'}

Someone has powered-down:

=====> withdraw_vesting
{'account': 'giveandtake1', 'vesting_shares': '1184211.402324 VESTS'}

Someone placed a conversion order:

=====> convert
{'amount': '0.158 SBD', 'owner': 'jamessmith', 'requestid': 1468935121}

This order executes 7 days after its initiated, and takes the 7 day average price as the execution basis. It is useful for placing a big order in the internal market without moving it too much.

Mining, Witnesses

Finding a pow (mining):

=====> pow
{'block_id': '0032d98aaa917775f98859c3235f43286479e1b7',
 'nonce': '6342739805603833088',
 'props': {'account_creation_fee': '0.001 STEEM',
           'maximum_block_size': 131072,
           'sbd_interest_rate': 1000},
 'work': {'input': '507906ad12f0d3bc57f636849dc117c5c970e79f0411e7e07459f78c09b350a0',
          'signature': '209c919b436563ae30171356cb7fcb6677794c9cbf163c695d3ce56841a9c86f1d06ab7f4b28c4c493a5d951664b32b7fb76714b30e49c8035bf12c520b1c6c3df',
          'work': '0000000350c2f2cae73dab00d1c2a1e2fafbf22e1e2fe0bb2a0a4e3a69c3b976',
          'worker': 'STM574Rx1dML1K8Mi9Uz42yWDYAr6ymrxC723HR1YX81Jqk7MSaRT'},
 'worker_account': 'redrockmining'}

Witness changes its settings:

=====> witness_update
{'block_signing_key': 'STM7yFmwPSKUP7FCV7Ut9Aev5cwfDzJZixcreS1U3ha36XG47ZpqT',
'fee': '0.000 STEEM',
'owner': 'anyx',
'props': {'account_creation_fee': '2.600 STEEM',
          'maximum_block_size': 131072,
          'sbd_interest_rate': 1000},
'url': ''}

Witness publishes new market rates:

=====> feed_publish
{'exchange_rate': {'base': '4.100 SBD', 'quote': '1.000 STEEM'},
'publisher': 'steemed'}


Place internal market order:

=====> limit_order_create
{'amount_to_sell': '94.000 STEEM',
 'expiration': '2016-07-19T16:10:40',
 'fill_or_kill': False,
 'min_to_receive': '260.000 SBD',
 'orderid': 94,
 'owner': 'ledzeppelin'}

Cancel internal market order:

=====> limit_order_cancel
{'orderid': 1468934907, 'owner': 'fabio'}

Account Sundry

Voting for a witness:

=====> account_witness_vote
{'account': 'leontyashka', 'approve': True, 'witness': 'arhag'}

Initiating account recovery:

=====> request_account_recovery
{'account_to_recover': 'stan',
 'extensions': [],
 'new_owner_authority': {'account_auths': [],
                         'key_auths': [['STM53xgdjGJ26QEF8qUbGsxguRBQU1UMqLKTT3LJ2ysgsLdr3zJk2',
                         'weight_threshold': 1},
 'recovery_account': 'steem'}

Account recovery complete:

=====> recover_account
{'account_to_recover': 'val',
 'extensions': [],
 'new_owner_authority': {'account_auths': [],
                         'key_auths': [['STM7igHgvdVmUJFJadc56d3wuk28xPo9NV4qQLhsHBzmpiQvTq2Pe',
                         'weight_threshold': 1},
 'recent_owner_authority': {'account_auths': [],
                            'key_auths': [['STM4wo7NZSUY86RDmaZrQuhuAybyCNSEfnpN2fHmKX7V4mQtPrsuK',
                            'weight_threshold': 1}}


Custom json currently only supports the follow plugin. More features to come in the future.

=====> custom_json
{'id': 'follow',
 'json': '{"follower":"manugbr93","following":"natenvos","what":["blog"]}',
 'required_auths': [],
 'required_posting_auths': ['manugbr93']}

what can be either "blog","posts", "blog", "posts" or "".
I suspect "" means unfollow. I do not know what is the difference between blog and posts however.

An finally, this mysterious custom operation:

=====> custom
{'data': '046162696408637563697374696d03c6278df998dd7cf0bac77dc3af89debf1f628eebbe9e86daa9762b7590630218023560f4ac629975cda967a694a07ebea5d912fcc31bf732aca9908ec9c04d603850ae00e0fd3705003f8ea76c2073f7cac9ce38f97dd8d32ba5df47adbea29c29758d776467ce16f055a5095563',
 'id': 777,
 'required_auths': ['abid']}

@jl777 thinks its an encrypted private message.

Example: Parsing the Blockchain

Suppose we want to find out how many followers we have, and get the list of people whom are following us.

Here is how we can parse each block, only looking for custom_json operations to find out:

from steemapi.steemnoderpc import SteemNodeRPC

rpc = SteemNodeRPC("ws://", "", "")

MY_USERNAME = "furion"

props = rpc.get_dynamic_global_properties()
current_block = 0
last_confirmed_block = props['last_irreversible_block_num']

while current_block < last_confirmed_block:
    current_block += 1
    block = rpc.get_block(current_block)
    if "transactions" in block:
        for tx in block["transactions"]:
            for opObj in tx["operations"]:
                #: Each operation is an array of the form
                #:    [type, {data}]
                op_type = opObj[0]
                op = opObj[1]

                if op_type == "custom_json" and op['id'] == "follow":
                    j = json.loads(op['json'])
                    if j['following'] == MY_USERNAME:
                        follower_count += 1
                        print("%s is following my %s, I have %d followers now." % (
                            j["follower"], j['what'], follower_count

I only have 8 followers, which makes me sad:

kilrathi is following my ['blog'], I have 1 followers now.
cloveandcinnamon is following my ['blog'], I have 2 followers now.
dcsignals is following my ['blog'], I have 3 followers now.
jerome-colley is following my ['blog'], I have 4 followers now.
mihserf is following my ['blog'], I have 5 followers now.
shaheer001 is following my ['blog'], I have 6 followers now.
helikopterben is following my ['blog'], I have 7 followers now.
carmasleeper is following my ['blog'], I have 8 followers now.

So if you liked this guide, and you'd like to see more, you know what to do.

Thank you for making awesome apps for Steem. I hope you have found my guide useful.
#steem #dev #steemit-dev #development #programming #python


limit_order_create, limit_order_cancel

How does one see executions?

Account history for an account has virtual operations for orders filled by that account. Currently the only way to see virtual ops is in account history, but I'm thinking of adding an event interface for them one of these days and writing a plugin to expose them over an API.


That would be spendid


This is a very good question. I think it may be under transfers. I am looking into it now.

I've just re-run the script, and it looks like I missed 1 operation type :

=====> set_withdraw_vesting_route
{'auto_vest': False,
 'from_account': 'steemroller',
 'percent': 100,
 'to_account': 'itsascam'}

there is a fill_order event in the array returned by get_history:
sprintf(params,"{"id":%llu,"method":"get_account_history","params":["%s", %d, %d]}",

 fill_order ({"current_owner":"enki","current_orderid":3402053187,"current_pays":"19.613 SBD","open_owner":"taker","open_orderid":1469136521,"open_pays":"5.792 STEEM"})

it shows both orderids and how much was filled

I really need to fix my steemd/cli_wallet install, I'm missing out on all the good stuff.

Finally, cli_wallet works. Had to do sudo ufw disable && sudo reboot.

Wow! This post has a lot of useful information for someone wanting to get into developing apps for this platform. Thanks for putting this together!

Thanks for the invaluable post, bookmarked for when I do set up myself a node :)

I'm not smart enough to make heads or tails of it, but I can see this will be invaluable for developers. :)

Thank you liberosist. Its actually really simple, just a lot of information. I wanted to make it convenient for all devs who want a 1 stop guide.

I know. It's just that I'm programming illiterate. So it's all Greek to me. :)

holy. Thank you. please more like this. I hope this gets upvoted!

I hope so too, but if it just helps a bunch of people I'm still happy.

Awesome write up! This clears up a few things I have been tinkering with. Keep up the good work.

Hi @furion thanks for the informative post. I'm a developer looking into getting started with building Steem apps on sidechains. Could you also provide a brief tutorial on that?

Thanks :D

Coin Marketplace

STEEM 0.20
TRX 0.12
JST 0.028
BTC 64275.63
ETH 3502.79
USDT 1.00
SBD 2.51