The practice of programming using Graphene-based blockchains

Good day!

As part of the work on Steepshot, a lot of useful information about working with Graphene-based blockchain APIs (such as Steem and Golos) has been accumulated. I decided to combine everything I’ve found and accumulate it in one article. I think this information will be useful to developers who want to integrate their applications with Graphene-based blockchains, especially assuming the documentation will appear only by the beginning of 2018 (see Steemit roadmap, visual roadmap).

The first thing to start with is to determine the parameters of the network to which we connect.

Network parameters

WebSocket

All interaction between the client and the server occurs through the use of web socket technology + JSON-RPC is used as a wrapper (https://en.wikipedia.org/wiki/JSON-RPC). Combined, this gives us the possibility of asynchronous interaction. Both technologies are well described and implemented in most popular languages.

Addresses for connection:

• Steem - wss://steemd.steemit.com
• Golos - wss://ws.golos.io

chain_id

Each blockchain network has its own unique 64-byte key. It is transmitted with every transaction to the network and is included in the encrypted message. Thus, the network is protected from accidental transactions.

Network Identifiers:

• Steem 0000000000000000000000000000000000000000000000000000000000000000
• Golos 782a3039b478c839e4cb0c941ff4eaeb7df40bdd68bd441afd444b9da763de12

Symbolic notation

In addition to the keys, the blockchains usually differ in their symbolic notation.

Working Name	Steemit	Golos
prefix	STM	GLS
steem_symbol	STEEM	GOLOS
sbd_symbol	SBD	GBG
vests_symbol	VESTS **1 **2	GESTS

They are also used in transactions when money values are being transferred. Therefore, for languages that do not support this type (like C#), I recommend writing your own implementation (apples and pears separately).

In addition, to avoid errors in the processing of dates and numbers associated with globalization settings in Steepshot, additional parameters were immediately added:

• CultureInfo - Standard C# class for managing globalization parameters (necessary for correct processing of dates, time, numbers)
• JsonSerializerSettings - the same as CultureInfo, is only used in the third-party library Newtonsoft.Json for parsing responses from blockchains.

Networking

All requests to the network can be divided into 2 types:

GET requests are the requests that do not change the network status (read-only). Requests can take place in an arbitrary form and do not require signatures from anyone, even from the browser. By following those links you can see a description of some GET requests as well as a little information about the message format sent and received:

• https://steemit.com/steem/@klye/an-introduction-to-steemd-api-calls-functions-and-usage
• https://golos.io/ru--otkrytyij-kod/@asuleymanov/opisanie-api-golos-chast-1
• https://steemit.github.io/steemit-docs
• https://steemdb.com
• https://steemd.com

POST requests are the requests that make changes in the network. They should be issued in the form of a transaction and signed by a private user key.

Let's consider in more detail what a transaction is and how to create it. A good example of how to build a transaction is described in the article from @xeroc: https://steemit.com/steem/@xeroc/steem-transaction-signing-in-a-nutshell. In order not to repeat the information from the article I will briefly describe the main points and add examples of the construction of various operations.

So, what is a transaction

transaction - the minimum indivisible unit of information sent by the client to the server in order to add the data in the block.

The transaction contains the required fields:

• ref_block_num - block number, unsigned number of 16 bytes.
• ref_block_prefix is a block prefix, an unsigned number of 32 bytes.
• expiration - the lifetime of the transaction (usually 30 seconds from the time the transaction was generated).
• operations - an array of the operations to be sent.
• extensions - array of possible additional parameters.
• signatures - an array of signatures.

How to get data to fill fields

• Before making transactions, you must make a request to the server.

Here's how the raw request looks through the socket:

  {"Method": "get_dynamic_global_properties", "params": [], "jsonrpc": "2.0", "id": 0}

In response, a block of data about the current state of the network will be returned:

{
  "id": 0,
  "head_block_number": 13506599,
  "head_block_id": "00ce18271e38c48379c4744702be5202d42b2d23",
  "time": "2017-07-08T15:23:09",
  "current_witness": "clayop",
  "total_pow": 514415,
  "num_pow_witnesses": 172,
  "virtual_supply": "253041799.029 STEEM",
  "current_supply": "251230822.919 STEEM",
  "confidential_supply": "0.000 STEEM",
  "current_sbd_supply": "3276055.783 SBD",
  "confidential_sbd_supply": "0.000 SBD",
  "total_vesting_fund_steem": "179261723.004 STEEM",
  "total_vesting_shares": "370713143905.498356 VESTS",
  "total_reward_fund_steem": "0.000 STEEM",
  "total_reward_shares2": "0",
  "pending_rewarded_vesting_shares": "226872178.104164 VESTS",
  "pending_rewarded_vesting_steem": "109617.757 STEEM",
  "sbd_interest_rate": 0,
  "sbd_print_rate": 10000,
  "average_block_size": 7086,
  "maximum_block_size": 65536,
  "current_aslot": 13564063,
  "recent_slots_filled": "340282366920938463463374607431768211455",
  "participation_count": 128,
  "last_irreversible_block_num": 13506579,
  "max_virtual_bandwidth": "5986734968066277376",
  "current_reserve_ratio": 20000,
  "vote_power_reserve_rate": 10
}

Based on this set, fill in the necessary fields for your transaction:

ref_block_num = head_block_number & 0xffff = 6183 (13506599 = 0xCE1827 take the lower 4 bytes (& 0xffff) we get 0x1827 = 6183
ref_block_prefix = head_block_id (take the lower bytes from 12 to 15 and translate to number) = 2210674718

(We translate a string into an array of bytes 0x00ce18271e38c48379c4744702be5202d42b2d23 and take the lower bytes from 12 to 15. 0x1e38c483 = 2210674718)

expiration = time + 30 sec. = "2017-07-08T15:23:39"

The remaining parameters are filled with user data.

operations

There are many types of operations. The full list can be found here: https://github.com/steemit/steem/blob/master/libraries/protocol/include/steemit/protocol/operations.hpp.
As you can see from the file static_variant is an enumeration. So each operation has its own serial number. This is important because it participates in the formation of a transaction signature.
Because of a large number of operations, a description of all of them (as well as a description of their fields) is beyond the scope of this article.

Transaction serialization

The transaction then is encrypted with a private key of the user, but before that, it is necessary to generate the encrypted message itself (which may not be such a simple matter).
As an example, consider a transaction adding a beneficiary. Here’s an example of a raw socket request:

{
 "method": "call",
 "params":
   [ 3,
     "broadcast_transaction",
     {
      "ref_block_num": 34294,
      "ref_block_prefix": 3707022213,
      "expiration": "2016-04-06T08:29:27",
      "operations":
       [
        [
         "comment_options",
         {
          "author": "author_test7",
          "permlink": "permlink_test8",
          "max_accepted_payout": "1000000.000 SBD",
          "percent_steem_dollars": 10000,
          "allow_votes": true,
          "allow_curation_rewards": true,
          "extensions":
           [
            [
             0,
             { "beneficiaries":
                [
                 {
                  "account": "account_test9",
                  "weight": 2000
                 },
                 {
                  "account": "account_test10",
                  "weight": 5000
                 }
                ]
             }
            ]
           ]
         }
        ]
       ],
      "extensions": [],
      "signatures": ["***********************************"]
     }
   ],
 "jsonrpc": "2.0",
 "id": 0
}

The example is long, but it shows the serialization of the main types well.
And here is a surprise. In serialized form, the message looks like a sequence of bytes:

0000000000000000000000000000000000000000000000000000000000000000f68585abf4dce7c8045701130c617574686f725f74657374370e7065726d6c696e6b5f746573743800ca9a3b000000000353424400000000102701010100020d6163636f756e745f7465737439d0070e6163636f756e745f746573743130881300

64 bytes of zeros in front of the message is nothing more than a network key (chain_id). It's not in JSON, but it is involved in encryption.

For more clarity, we split the message into components:

Ok so, what's going on here?
As you can see, the field names aren’t used in the formation of posts, which means that you can not change their order.
All the values are serialized according to their data type:

Type	Value
bool	1 byte
byte	As is (1 byte)
Int16 / UInt16	2 byte
Int32 / Uint32 / float	4 byte
Int64 / Uint64 / double	8 byte
DateTime	It is taken as the number of ticks since 01/01/1970 4 byte
Array ***	The output should be an array of bytes consisting of a prefix (the size of the array) and the array of the message. See details under table ***.
String	1. The string must be converted to UTF8 format after that into bytes. In C#, this can be done with the single command Encoding.UTF8.GetBytes. 2. The resulting array is processed as described in the recipe for Array.
Money ("1000000.000 SBD")	It consists of 1) values ​of 1000000000 - 8 bytes, 2) accuracy order 3 - 1 byte, 3) currency name - 7 bytes (Unlike string encryption, the length of the word is not specified, instead the size is reserved in advance in the amount of 7 bytes From the length of the name). The total is 16 bytes.
Operation and compound types of objects	All compound objects are written as a sequence of simple type fields. As a special case, you can mark operation objects, besides the fields (displayed in JSON) themselves, they contain a field of operation type (for specific values ​​see here (https://github.com/steemit/steem/blob/master/libraries/protocol/include/steemit/protocol/operations.hpp) which is used only for serialization.

*** Array
For this:
1.) We get the size of the array, translate the resulting number into bytes and write it into the output array. To translate a number into bytes, use the function:

def varint(n):
 """ Varint encoding """
 data = b''
 while n >= 0x80:
   data += bytes([(n & 0x7f) | 0x80])
   n >>= 7
 data += bytes([n])
 return data

The source: https://github.com/xeroc/python-graphenelib/blob/master/graphenebase/types.py

2.) Sequentially translate and add to the output array all the elements of the input array

Creating a signature

The final part of the transaction processing is the creation of a signature on the received serialized message.
By the signature, we mean a unique byte array that was obtained using the encryption algorithm. In our case ECDSA (Elliptic Curve Digital Signature Algorithm) called Secp256k1 is used.

There are several existing implementations of this algorithm:

• https://github.com/sipa/secp256k1 – the fastest one, written in C
• http://www.bouncycastle.org
• https://github.com/Chainers/Cryptography.ECDSA
• https://github.com/warner/python-ecdsa

As a rule, it is not the message but its hash that is submitted for signing. So for Steem, SHA256 is used. And for the Bitcoin SHA256 is applied even twice in a row ...
The received signature (or the signature, if it is necessary to use several user keys) is added to the "signatures" field of the output message.
On this process the formation of the transaction can be considered complete, it remains to transfer it to the server.

Separately, note that in Steem API there is a method verify_authority, it is convenient to use it to verify the implementation of the transaction code and the validation of the transaction signature without directly adding it to the block. This can be useful for compiling automated tests.

.NET dev status

Lib name	Description	Act. version
Cryptography.ECDSA	Implementation of ECDSA for transaction signing	2.0
Ditch	Create and broadcast transactions to blockchain	2.1.2

Team progress on .NET encryption & signing

• C# library Ditch 2.0 for Graphene
• C# library Cryptography.ECDSA 2.1 for Graphene
• (ANN) C# library Ditch 1.0 for Graphene
• (ANN) C# library Cryptography.ECDSA 1.0 for Graphene

Discuss

in Telegram
in Steemit.chat
email us