Caution When Requesting Data From Steem RPC Nodes
![](https://steemitimages.com/640x0/https://cdn.steemitimages.com/DQmYHDjBE78RArUadjW1bDFNsb5XoGzqxfSiFrBTWoEqBF3/woman-3597095_1920.jpg)
If you are a developer and you are working with data from public RPC endpoints, this might be of interest for you. I've found the reason for the duplicate account operations and posts on SteemWorld and just want to inform you about the cause for the issues here.
Don't use the field 'id'
When you make a get_content
call or any other that returns posts, the returned post-id can differ from node to node. I recommend to use the fields author
and permlink
for identifying posts and never rely on a (global) unique ID in the field id
, because otherwise you will get in trouble for sure.
The same is true for loading data from the account history with get_account_history
. The returned id
might not always be the same. Even when using just one endpoint like api.steemit.com
, the returned operations must not always have the same id
and, as I saw in my tests, they can even differ from minute to minute. The reason for this will be explained below.
How to identify one unique account operation
A few months ago I changed my getAccountHistory
method on SteemWorld, so that each operation contains an unique_id
. The only way to achieve this is to build an own ID by combining the block
, trx_id
or timestamp
with the operation's content. One can't rely on the transaction details like op_in_trx
or trx_in_block
solely, because they can differ with different node configurations.
I'm building the ID by combining the block-number with the op-type in op[0]
and a 32-bit hash of the operation's data in op[1]
:
buildUniqueId: function(blockNum, op)
{
let i, key, hash,
data = {},
keys = Object.keys( op[1] ).sort();
for (i=0; i<keys.length; i++) {
key = keys[i];
data[ key ] = op[1][ key ];
}
hash = fastHash( JSON.stringify(data) );
return blockNum + "-" + op[0] + "-" + hash;
}
Sometimes there were still duplicate operations coming up in the past weeks and the reason for this was the field order in op[1]
, which can also differ from node to node. Therefore I'm sorting the fields by name first (as shown in the code above) and it works without any issues now.
Of course, a fast 32-bit hash may not be the right choice when storing the operations in a database, but for my purpose on SteemWorld it is good enough and does the job very well.
Keep in mind that your requests might be routed to different endpoints
Some RPC node maintainers do route incoming requests to other endpoints, when they make a replay
or change configurations on their servers, therefore you can't say for sure that the received data always come from the same node. That is why you should never make use of the field id
to get a (global) unique ID for the received data.
Hope this helps some of you.
Danke für den Tipp!!!!
That seems like a huge bug. If a field is useless because it cannot be relied upon, then it should not exist.
It's not really a bug, because the field just contains an index for the data and the nodes can be configured with different plugins and settings. When a node does not hold all types of operations for example, the id will of course not be the same as for other nodes.
The field can only be used for local/private environments and even then I would be cautious. It's a bit irritating for developers, but I'm glad that I found a solution now :)
Yes. It's a common solution for certain things...just usually not when you're asking something of a database... Very good to know for developers though.
This post has been just added as new item to timeline of SteemWorld on Steem Projects.
If you want to be notified about new updates from this project, register on Steem Projects and add SteemWorld to your favorite projects.
This post has been included in the latest edition of SOS Daily News - a digest of all you need to know about the State of Steem.
Editor of the The State of Steem SoS Daily News.
Promoter of The State of Steem SoS Weekly Forums.
Editor of the weekly listing of steem radio shows, podcasts & social broadcasts.
Founder of the A Dollar A Day charitable giving project.
Hy steemchiller
Did you know that you have 70493 transactions on steem blockchain!
You can check and download all your transactions, votes, comments and more with @steemdetective!
Start here steemdetective.com
Hi @steemchiller!
Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your UA account score is currently 6.145 which ranks you at #263 across all Steem accounts.
Your rank has improved 3 places in the last three days (old rank 266).
In our last Algorithmic Curation Round, consisting of 249 contributions, your post is ranked at #14.
Evaluation of your UA score:
Feel free to join our @steem-ua Discord server