Developers Guide to Steem's Blockchain

in #steem4 years ago

I have been building a blockchain parser for my Steem Fundamental Analysis project, and I thought it would be a good idea to document my findings.

If you're a developer and you'd like to build an app on top of Steem/Steemit, this guide is for you.

Setting up your node

First, we need to setup a node. There are plenty of guides and tools (for Docker, Ubuntu, Windows) on steemit on how to do that, so I won't repeat myself here.

To get access to all the API's, modify your config.ini to have the following values:

enable-plugin = witness account_history tags follow market_history
public-api = database_api login_api network_broadcast_api follow_api market_history_api

Also, you should remove miner and witness fields. We aren't interested in mining, just running a node.

Now you can run steemd:

./steemd --rpc-endpoint

The --rpc-endpoint will setup a websocket endpoint at ws://127.0.0.1:8090.

You should be able to connect to it from your favorite programming language (Python, JavaScript, etc.), or by using a tool such as wssh.


If you're using Python, then I suggest the awesome python-steemlib by @xeroc.

You can connect to the RPC like so:

from steemapi.steemnoderpc import SteemNodeRPC

rpc = SteemNodeRPC("ws://127.0.0.1:8090", "", "")

Analyzing the Blockchain

Now that we have a node running, and we are connected to it, we can issue commands.

In this tutorial, we will be using the following RPC commands:

get_block()
get_dynamic_global_properties()

Again, the RPC is available for us at ws://localhost:9080.


The full list of commands is available here (scroll to the bottom):
https://github.com/steemit/steem/blob/master/libraries/app/include/steemit/app/database_api.hpp


In order to get a block from the blockchain, we want to call a RPC command called get_block, which takes just one argument: Block ID.

Block ID is just a number, anywhere between 0 and the current block ID.
Each new block has an ID at an increment of 1.

You can find out what the current block ID is by looking at steemd output:

651068ms th_a       application.cpp:439           handle_block         ] Got 1 transactions from network on block 3695832

Or by asking steemd via RPC:

 props = rpc.get_dynamic_global_properties()
 last_confirmed_block = props['last_irreversible_block_num']

So to traverse the entire blockchain, from start to finish, we would do something like this:

props = rpc.get_dynamic_global_properties()
current_block = 0
last_confirmed_block = props['last_irreversible_block_num']

while current_block < last_confirmed_block:
    current_block += 1
    print("Processing block %d" % current_block)

    block = rpc.get_block(current_block)
    pprint(block)

Now lets look at what get_block gives us. If we call rpc.get_block(3332318), we get a block 3332318 in a json format, that looks like this:

{'extensions': [],
 'previous': '0032d8dd7fef5c0eafd7667805134effb9ef21e9',
 'timestamp': '2016-07-19T13:29:15',
 'transaction_merkle_root': '5be8c3f95a29253153e911fd079528ad8216958a',
 'transactions': [{'expiration': '2016-07-19T13:29:24',
                   'extensions': [],
                   'operations': [['vote',
                                   {'author': 'beowulfoflegend',
                                    'permlink': 'how-steemit-self-polices-content',
                                    'voter': 'jills',
                                    'weight': 10000}]],
                   'ref_block_num': 55512,
                   'ref_block_prefix': 2485993294,
                   'signatures': ['20196c89f5e2ca58da3377a30ee08782df59dbd4b72db40ec3348f3417ff653fa84cf684748853e8ea43363c747c74b4ff015626ef9e2b2e902e0ee98a80195336']},
                  {'expiration': '2016-07-19T13:30:10',
                   'extensions': [],
                   'operations': [['comment',
                                   {'author': 'weenis',
                                    'body': 'This is Amazing! Upvoted! \n',
                                    'json_metadata': '',
                                    'parent_author': 'begstreets',
                                    'parent_permlink': 'abstractions-no-1',
                                    'permlink': 'abstractions-no-1',
                                    'title': ''}]],
                   'ref_block_num': 55516,
                   'ref_block_prefix': 765250389,
                   'signatures': ['1f7711cfc441bbc5e6df16d7b73ea7d4ea7d7a40a3c3a1fd857d67b50b30c139d06f742e5fd15a68f2109469c977a12c50abab81632b8a4ba5e328c5048290153b']},
 ],
 'witness': 'riverhead',
 'witness_signature': '1f4236f6a0ff79ab2ff9a190162aeb25b5e1509ca44df8803c8091a2abcfe2022f733eb77c15ec8265a1b1d0d2226038b3259db9ebe17b871bcbf3be0dee605bc9'}

We are mostly interested in transactions and operations. Transactions contains a list of operations, which are basically all the events that happened on the blockchain within the 3 second period included in the block.

Operation Types

Currently, the Steem blockchain contains the following operations:

  • comment, delete_comment, vote
  • account_create, account_update, request_account_recovery, recover_account
  • limit_order_create, limit_order_cancel
  • transfer, transfer_to_vesting, withdraw_vesting, convert
  • pow, feed_publish, witness_update, account_witness_vote
  • custom, custom_json

Blog Posts, Comments and Votes

This is a blog post:

=====> comment
{'author': 'rozu15',
 'body': 'content omited becuse it breaks my post',
 'json_metadata': '{"tags":["trump"],"links":["https://twitter.com/realDonaldTrump/status/755254384062263296"]}',
 'parent_author': '',
 'parent_permlink': 'trump',
 'permlink': 'us-election-melania-trump-plagiarised-michelle-obama',
 'title': "US election: Melania Trump 'plagiarised' Michelle Obama"}

This is a comment. It has empty title:

=====> comment
{'author': 'densmirnov',
'body': 'nice nickname',
'json_metadata': '{"tags":["steem"]}',
'parent_author': 'steemiscrap',
'parent_permlink': 're-cryptorune-steem-dollar-poloniex-pump-is-a-arbitrage-goldmine-20160719t133438897z',
'permlink': 're-steemiscrap-re-cryptorune-steem-dollar-poloniex-pump-is-a-arbitrage-goldmine-20160719t133602671z',
'title': ''}

The comment has been deleted:

=====> delete_comment
{'author': 'densmirnov',
'permlink': 're-cryptorune-re-densmirnov-re-cryptorune-steem-dollar-poloniex-pump-is-a-arbitrage-goldmine-20160719t133207807z'}

The comment has received a vote:

=====> vote
{'author': 'kain-jc',
 'permlink': 'a-meditation-on-love',
 'voter': 'cantinhodatete',
 'weight': 10000}

If we divide the weight by 100, we get the % of the vote. For instance, a full upvote is 100%. The full downvote is -100%. Half the upvote is 50%.

Account Features

Account has been created:

=====> account_create
{'active': {'account_auths': [],
            'key_auths': [['STM7huHJRrqHE6FDNHJEsUYNNKuQKBfr4iS3xwgyGzXDurbiCRx2V',
                           1]],
            'weight_threshold': 1},
 'creator': 'steem',
 'fee': '5.000 STEEM',
 'json_metadata': '',
 'memo_key': 'STM7R3iScNyQCytvB8EcRdEYLfJCGp3UkyNm7Mt4c9BE1ZoUTdazw',
 'new_account_name': 'saripdol',
 'owner': {'account_auths': [],
           'key_auths': [['STM8XzPFfu3yCujJiSbgdJvKm69LMcm1ckocuvoSaaWRmPRzfH7jr',
                          1]],
           'weight_threshold': 1},
 'posting': {'account_auths': [],
             'key_auths': [['STM8KKqZufL2PM7V9dHeSW8KpN4wWs8cGaiQsz7FCrPohX1ZkfPvv',
                            1]],
             'weight_threshold': 1}}

Account has been updated:

=====> account_update
{'account': 'signaltonoise',
 'active': {'account_auths': [],
            'key_auths': [['STM83ihutAefZSHCJRXLHiRnji8VhQHcouMJc6xGtaLW6PCARXiDa',
                           1]],
            'weight_threshold': 1},
 'json_metadata': '',
 'memo_key': 'STM81hFsfm2csFyPYDCdwjuGJX33EB2Dx6rG74AfFgStu9C6PvzMd',
 'owner': {'account_auths': [],
           'key_auths': [['STM8fcgjrpKbVLHyy3RKnQGx3dhYEdY7bKhbv3FaML5ooK8PpdJaR',
                          1]],
           'weight_threshold': 1},
 'posting': {'account_auths': [],
             'key_auths': [['STM75QK5e69SAuMzVtJqGex3HcaqrPcxpXgxb72WxbUJn2apRX7KX',
                            1]],
             'weight_threshold': 1}}

Transfers, Vesting, Conversions

A transfer of SBD or STEEM from one account to another.

=====> transfer
{'amount': '34.428 SBD',
 'from': 'thisischris225',
 'memo': 'ee1d91d4036200c7',
 'to': 'poloniex'}

Someone has powered-up:

=====> transfer_to_vesting
{'amount': '11.789 STEEM', 'from': 'leksimus', 'to': 'leksimus'}

Someone has powered-down:

=====> withdraw_vesting
{'account': 'giveandtake1', 'vesting_shares': '1184211.402324 VESTS'}

Someone placed a conversion order:

=====> convert
{'amount': '0.158 SBD', 'owner': 'jamessmith', 'requestid': 1468935121}

This order executes 7 days after its initiated, and takes the 7 day average price as the execution basis. It is useful for placing a big order in the internal market without moving it too much.

Mining, Witnesses

Finding a pow (mining):

=====> pow
{'block_id': '0032d98aaa917775f98859c3235f43286479e1b7',
 'nonce': '6342739805603833088',
 'props': {'account_creation_fee': '0.001 STEEM',
           'maximum_block_size': 131072,
           'sbd_interest_rate': 1000},
 'work': {'input': '507906ad12f0d3bc57f636849dc117c5c970e79f0411e7e07459f78c09b350a0',
          'signature': '209c919b436563ae30171356cb7fcb6677794c9cbf163c695d3ce56841a9c86f1d06ab7f4b28c4c493a5d951664b32b7fb76714b30e49c8035bf12c520b1c6c3df',
          'work': '0000000350c2f2cae73dab00d1c2a1e2fafbf22e1e2fe0bb2a0a4e3a69c3b976',
          'worker': 'STM574Rx1dML1K8Mi9Uz42yWDYAr6ymrxC723HR1YX81Jqk7MSaRT'},
 'worker_account': 'redrockmining'}

Witness changes its settings:

=====> witness_update
{'block_signing_key': 'STM7yFmwPSKUP7FCV7Ut9Aev5cwfDzJZixcreS1U3ha36XG47ZpqT',
'fee': '0.000 STEEM',
'owner': 'anyx',
'props': {'account_creation_fee': '2.600 STEEM',
          'maximum_block_size': 131072,
          'sbd_interest_rate': 1000},
'url': 'https://steemit.com/witness-category/@anyx/hello-from-anyx'}

Witness publishes new market rates:

=====> feed_publish
{'exchange_rate': {'base': '4.100 SBD', 'quote': '1.000 STEEM'},
'publisher': 'steemed'}

Trading

Place internal market order:

=====> limit_order_create
{'amount_to_sell': '94.000 STEEM',
 'expiration': '2016-07-19T16:10:40',
 'fill_or_kill': False,
 'min_to_receive': '260.000 SBD',
 'orderid': 94,
 'owner': 'ledzeppelin'}

Cancel internal market order:

=====> limit_order_cancel
{'orderid': 1468934907, 'owner': 'fabio'}

Account Sundry

Voting for a witness:

=====> account_witness_vote
{'account': 'leontyashka', 'approve': True, 'witness': 'arhag'}

Initiating account recovery:

=====> request_account_recovery
{'account_to_recover': 'stan',
 'extensions': [],
 'new_owner_authority': {'account_auths': [],
                         'key_auths': [['STM53xgdjGJ26QEF8qUbGsxguRBQU1UMqLKTT3LJ2ysgsLdr3zJk2',
                                        1]],
                         'weight_threshold': 1},
 'recovery_account': 'steem'}

Account recovery complete:

=====> recover_account
{'account_to_recover': 'val',
 'extensions': [],
 'new_owner_authority': {'account_auths': [],
                         'key_auths': [['STM7igHgvdVmUJFJadc56d3wuk28xPo9NV4qQLhsHBzmpiQvTq2Pe',
                                        1]],
                         'weight_threshold': 1},
 'recent_owner_authority': {'account_auths': [],
                            'key_auths': [['STM4wo7NZSUY86RDmaZrQuhuAybyCNSEfnpN2fHmKX7V4mQtPrsuK',
                                           1]],
                            'weight_threshold': 1}}

Custom

Custom json currently only supports the follow plugin. More features to come in the future.

=====> custom_json
{'id': 'follow',
 'json': '{"follower":"manugbr93","following":"natenvos","what":["blog"]}',
 'required_auths': [],
 'required_posting_auths': ['manugbr93']}

what can be either "blog","posts", "blog", "posts" or "".
I suspect "" means unfollow. I do not know what is the difference between blog and posts however.

An finally, this mysterious custom operation:

=====> custom
{'data': '046162696408637563697374696d03c6278df998dd7cf0bac77dc3af89debf1f628eebbe9e86daa9762b7590630218023560f4ac629975cda967a694a07ebea5d912fcc31bf732aca9908ec9c04d603850ae00e0fd3705003f8ea76c2073f7cac9ce38f97dd8d32ba5df47adbea29c29758d776467ce16f055a5095563',
 'id': 777,
 'required_auths': ['abid']}

@jl777 thinks its an encrypted private message.

Example: Parsing the Blockchain

Suppose we want to find out how many followers we have, and get the list of people whom are following us.

Here is how we can parse each block, only looking for custom_json operations to find out:

from steemapi.steemnoderpc import SteemNodeRPC

rpc = SteemNodeRPC("ws://127.0.0.1:8090", "", "")

MY_USERNAME = "furion"

props = rpc.get_dynamic_global_properties()
current_block = 0
last_confirmed_block = props['last_irreversible_block_num']

while current_block < last_confirmed_block:
    current_block += 1
    block = rpc.get_block(current_block)
    
    if "transactions" in block:
        for tx in block["transactions"]:
            for opObj in tx["operations"]:
                #: Each operation is an array of the form
                #:    [type, {data}]
                op_type = opObj[0]
                op = opObj[1]

                if op_type == "custom_json" and op['id'] == "follow":
                    j = json.loads(op['json'])
                    if j['following'] == MY_USERNAME:
                        follower_count += 1
                        print("%s is following my %s, I have %d followers now." % (
                            j["follower"], j['what'], follower_count
                        ))
                        

I only have 8 followers, which makes me sad:

kilrathi is following my ['blog'], I have 1 followers now.
cloveandcinnamon is following my ['blog'], I have 2 followers now.
dcsignals is following my ['blog'], I have 3 followers now.
jerome-colley is following my ['blog'], I have 4 followers now.
mihserf is following my ['blog'], I have 5 followers now.
shaheer001 is following my ['blog'], I have 6 followers now.
helikopterben is following my ['blog'], I have 7 followers now.
carmasleeper is following my ['blog'], I have 8 followers now.

So if you liked this guide, and you'd like to see more, you know what to do.


Thank you for making awesome apps for Steem. I hope you have found my guide useful.
#steem #dev #steemit-dev #development #programming #python

Sort:  

limit_order_create, limit_order_cancel

How does one see executions?

Account history for an account has virtual operations for orders filled by that account. Currently the only way to see virtual ops is in account history, but I'm thinking of adding an event interface for them one of these days and writing a plugin to expose them over an API.

Loading...

That would be spendid

Loading...

This is a very good question. I think it may be under transfers. I am looking into it now.

I've just re-run the script, and it looks like I missed 1 operation type :

=====> set_withdraw_vesting_route
{'auto_vest': False,
 'from_account': 'steemroller',
 'percent': 100,
 'to_account': 'itsascam'}

there is a fill_order event in the array returned by get_history:
sprintf(url,"http://127.0.0.1:8090");
sprintf(params,"{"id":%llu,"method":"get_account_history","params":["%s", %d, %d]}",

 fill_order ({"current_owner":"enki","current_orderid":3402053187,"current_pays":"19.613 SBD","open_owner":"taker","open_orderid":1469136521,"open_pays":"5.792 STEEM"})

it shows both orderids and how much was filled

I really need to fix my steemd/cli_wallet install, I'm missing out on all the good stuff.

Finally, cli_wallet works. Had to do sudo ufw disable && sudo reboot.

Wow! This post has a lot of useful information for someone wanting to get into developing apps for this platform. Thanks for putting this together!

Thanks for the invaluable post, bookmarked for when I do set up myself a node :)

I'm not smart enough to make heads or tails of it, but I can see this will be invaluable for developers. :)

Thank you liberosist. Its actually really simple, just a lot of information. I wanted to make it convenient for all devs who want a 1 stop guide.

I know. It's just that I'm programming illiterate. So it's all Greek to me. :)

holy. Thank you. please more like this. I hope this gets upvoted!

I hope so too, but if it just helps a bunch of people I'm still happy.

Awesome write up! This clears up a few things I have been tinkering with. Keep up the good work.

Hi @furion thanks for the informative post. I'm a developer looking into getting started with building Steem apps on sidechains. Could you also provide a brief tutorial on that?

Thanks :D

Just noticed the library being used is Python3 :( Any idea why there are only Python3 libraries/apis being made?

Because after 10 years, everyone's (finally!) moving to Python 3.

Great stuff, Im not a developer but this post has so much value to everyone :) Thanks!

This is how to delete a post or comment programmatically

curl http://127.0.0.1:8093 --data '{"jsonrpc": "2.0", "method": "sign_transaction", "params": [{"operations": [ ["delete_comment", {"author": "username", "permlink": "re-to-some-timestamp-z", "required_posting_auths": ["username"]}] ] }, true], "id": 1}'

There is just one thing I didn't entirely get, if I run my own node, would it pull the blockchain from seed and make local copy of it or start over? Is this an isolated development environment I can use without risking creating real posts and spending real money?

Otherwise, thanks for good posts.

Great article, @furion. Thank you so much. So, I have to use the comment operation if I want to publish a new blog post on my blog. Is it correct?

👍great post.. @furion

The article goes right into the details! Thanks!

Thanks @furion for the comprehensive guide.

You have a typo in the port number:
Again, the RPC is available for us at ws://localhost:9080
should be 8090 I believe

Bookmarking, thanks! $b.programming $b.code $b.steem

Two thumbs Way up!

Nice one @furion. Thank you!

great effort !!! @furion :))

Fantastic. Thank you!

i followed , i might be ditching one of my two jobs so i can play with steemd/cli too

That is going to be very usefull ...

I also have put up a guide to create your own local steemit website, you can find it here (https://steemit.com/steemit/@artakan/how-to-build-your-own-steemit-com-website) and it's on line here

https://my-steemit.com / [https://my-steemit.com

It can be usefull when this happens
like now for example ;-)