Exploratory Analysis Shows World of Opportunity with JSON and STEEM

in utopian-io •  8 days ago

The use of Custom JSON field on the STEEM Blockchain is becoming more and more popular.  From discussion I have read, this field can do almost anything from creating Tokens to holding game information.

JSON data is a way of representing objects or arrays.  It is easy to read, and it is easy to parse.  Many API calls will return JSON format and many web apps use JSON to easily move information around the internet.

Its is therefore no surprise that developers are now using the JSON filed on steem to move data between an app and the blockchain.

Initially the Custom JSON field was used for holding follow information, where one account on the blockchain follows another.  However, over time, this has expanded to holding information from multiple sources for multiple reasons.

With that in mind I wanted to take a quick look at the Txcustoms table in the STEEMSQL database to just get an idea of what and who is using this JSON field.  This type of analysis is commonly called exploratory analysis.

Repository

https://github.com/steemit/steem

Aim of Exploratory Analysis

- Get a feel for the Txcustoms table in STEEMSQL.

- Establish if there was growth in the use of this field from 2018 to 2019

- Establish what app is using this most

- Get an idea of the type of data held 

Analysis

The table below shows the custom JSONs for the first week of January 2018. None of the Custom JSONs required additional posting or other authorities.  As we can see, most of the transitions using this field were Follows, however there were a few others too.

 

Pacman-live seems to be one of the first games to host data on the blockchain.  Looking at this further the data sent to the blockchain was the username, the score, the game level and a timestamp.

  

Chainbb seem to have been using the JSON field to store details of Forum objects, such as forum configurations and forum posts.

  

Distribution is an interesting one and seems to be holding information about distributions from randowhale

  

 And we can also see some testing of STEEMMonsters under TEST_sm_pre_sale_packs

  

Jumping on now to the first week of January 2019 and things are looking rather different. 980162 custom JSON fields were used, down 58% on the same time last year.  However, this is not bad news because in 2018 99.8% of these related to follows and unfollows.  With the downturn in activity on the blockchain, for the same period in 2019 follows and unfollows make up only 9.8% of the custom JSON transactions.  The was a reduction of 95% in the actual number of follows and unfollows.

  

The table above shows the number of transitions of each time with custom JSONS. I have split this list in two.  On the right we have all the STEEMMonsters custom JSONS.  This makes up 88% of all custom JSONs.  On the left we have others such as Follows, DiceHash and Dlux.

The table below shows a sample of the data contained within some of the JSONs.

 

Also changed considerably since 2018 is the user of authority to post and this is also recorded in the blockchain and the SQL database.  Looking at this for STEEMMonsters in the table below, by doing a count of the required_posting_auths we can see that for this period there were 366,192 find match and by carrying out a distinct count of the same field, we can establish this was by 2379 different accounts.

  

Conclusion

Although there was a reduction in the number of custom JSONs used from Jan 18 to Jan 19, this is a reflection on the downturn in social activity on the block and not development activity.

The social activity, follows and unfollows is down 95%.  However, in 2018 non-social activity recorded in JSON was only 0.02%, in 2018 this was 90.2%.  This is an indication of development on the block and a sign in the shift of the platform from being a content/social platform to something more.

By looking at the data contained within the JSONs my eyes have been opened on how simple it can be to really make ‘ANY’ app a DApp on the STEEM Blockchain and I hope it gives you that sense too.  

Obviously STEEMMonsters are well out there in terms of usage compared to other apps using JSONs but many apps are only being to test this feature.  Some of this comes from the shift away from Steemit.Inc as the only developers on the block.  STEEM is coded in C++ which is a skill many do not have.  However, there is no shortage of App developers with JSON knowledge.

Now that I have a better understanding of how this table in the SQL database is kept and what data it contains, it opens up a new world of analysis.

So, who will be first to have a D+ site ready to replace G+ communities using our new friend JSON?

Data and Query

The data for this post was taken from the STEEMSQL database held and managed by @arcange.

The query used for 2019 was

select *
from Txcustoms
where CONVERT(DATE,timestamp) BETWEEN '2019-01-01' AND '2019-01-07'

and for 2018 was

select *
from Txcustoms
where CONVERT(DATE,timestamp) BETWEEN '2018-01-01' AND '2018-01-07'

As always, I used PowerBI to

- Connect to the data

- Transform the data

- Carry out calculations

- Visualise the data

If you want to learn more about using PowerBI or Excel do follow @theexcelclub


Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Hi @paulag, great topic, the custom-JSONs can be used for a number of things, and have seen quite some popularity with several apps and projects on top of Steem. Steemmonsters and MagicDice are probably the most prominent use case as of now. Unfortunately, SteemSQL is not fully ready for analyzing this yet, so the senders of the ops are only partly available at the moment. I would have loved to see more data covered than two periods of one week each. I could imagine having maybe weekly counts for a couple of tids could have provided additional insights here?

Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.

To view those questions and the relevant answers related to your post, click here.


Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]

·

Thank you for your review, @crokkon! Keep up the good work!

Hello Paula,

Sorry to say it, but your analysis is (partially) invalid :/

The table below shows the custom JSONs for the first week of January 2018. None of the Custom JSONs required additional posting or other authorities.

There was a bug in SteemSQL Database Injector, which was not processing required_auths and required_posting_auths.
This problem was fixed in December 2018 and only new custom_json operations issued after the fix were recorded correctly.

Unfortunately, SteemSQL need to reprocess all blocks from the beginning of the blockchain, something I plan to do this month.

(upvoted for visibility)

·

Okay, thanks for letting us know. Good luck reprocessing, that will take ages.

·
·

·
·
·

lol love it, and thanks so much for adding extra value to this post in the comments, your a rock star

Cool.
Is there a record of who adds the custom JSON?
Any authorization to add a field that already exists and is being pulled for data?
Can anyone add anything, like any account add info for any JSON data and imitate an app, such as adding bogus data?
Could I just add followers to people who didn't really follow?
Can I add SM data just by adding a custom JSON object?

Thanks!

·

Can I add SM data just by adding a custom JSON object?

Yes and No. You can add any custom_json content, including similar objects as SM, but the author of the transaction won't match.

Check my comment to @paulag

·
·

Thanks. So if someone makes an app, they can just look for custom json where their app (or another) is the author of the data and only use that?

·
·
·

They need to look at the author of the custom_json, not "the app", unless you want to monitor all transactions written as "something you understand" (i.e. like how steemit.com works with "follow" custom_json).

·
·
·
·

Yes, the appname as the author, like busy, or partiko, etc. To get all real follows then, you need to search for all app accounts that add to follow? If you don't look for busy or partiko in the follow, you won't get all the followers, right?

·
·
·
·
·

Here's how I'd explain it: anyone can create a custom_json transaction with an id. This could be something like follow, sm_team_reveal, etc.

When someone creates a custom_json follow transaction with data krnel they express that they want to follow krnel. Then every Steem node sees that they expressed that they want to follow krnel, so they update their internal books that they followed krnel.

Same with sm_claim_reward. If someone broadcasts an sm_claim_reward, then everyone knows that they want to claim their reward, and it is considered that they have claimed their reward.

·
·
·
·
·
·

OK, so it's per user account. Nothing to do with the app account, for follows. Are there fields that are validated through an app account? As a way to prevent anyone from just adding any data they want.

Say you have an app and create a field for data, how can you prevent just anyone from adding data to the field simply by signing/broadcasting it with their own account? Anyone could make fraudulent data claims on the field in that case. Is this a limitation of the custom JSON field, where it cant be used like any database?

Thanks!

·
·
·
·
·
·
·

Is this a limitation of the custom JSON field, where it cant be used like any database?

The custom_json transaction is not in any way a database. All the custom_json transaction is is a way to have a transaction that doesn't have any side effects. An example of a similar thing in Bitcoin would be sending 0 BTC to 0x0 with a certain message. That's all a custom_json does. If someone creates a custom_json transaction transfer krnel 5 cats and that transaction is included in a block nothing happens. The Steem nodes don't add an entry to a database saying that now @krnel has 5 more cats or something. The only thing that happens is that now if you ask a Steem node what was in that block, you will see a bunch of transactions, one of them being a custom_json transaction transfer krnel 5 cats. I could create a custom_json transaction saying sdadg asdgsad sdfasd asdf and none of that would be entered in any database (not like there would be any reason to), but will be entered in a block.

So why would this be useful for DApp creation? All you can do is write a message in a transaction to the blockchain without any side effects. But what this means is that my Stratos DApp can look for when @alice creates a transfer_stratos custom_json transaction. Then it can rely on a shared protocol; everyone running Stratos agrees that when someone creates a transfer_stratos custom_json transaction, they decide that that means that they will update their internal ledger accordingly.

So let's say Alice creates a custom_json transaction transfer_stratos 10 bob (this means 'transfer 10 SRTS to Bob', but written in a way that's easier for computers to understand). That transaction is included in a block, but the Steem nodes don't do anything about it. Again, they don't update a database or anything. They just include that transaction, charge Resource Credits, and ignore it. But Alice and Bob and many other users in Stratos have agreed that when anyone creates a transfer_stratos custom_json transaction, they have an intent to transfer SRTS to someone else (again, this is just a shared belief and doesn't mean anything on the blockchain level). So when a Stratos node running the Stratos software (connected to a Steem node) sees the transfer_stratos transaction and updates the internal balances to remove 10 SRTS from Alice's balance and add 10 to Bob's balance. Now every Stratos node agrees on this new update, and Alice has successfully transferred 10 SRTS to Bob (this actually means that every Stratos node agrees that the transaction happened, and so they updated their ledger accordingly).

Here's a post where I explain the same thing in more detail, including cases of malicious actors trying to hack the Stratos network, and explanations of how this relates to Steem Monsters and steem-state (note that this is before the rebrand from Engine to Stratos): https://steemit.com/steem/@shredz7/how-engine-works.

I'm glad to see you interested in this as it is a fascinating topic -- if you have any more questions feel free to ask!

·

Great questions

"Is there a record of who adds the custom JSON?"
In that table, not really, but there is posting auth recorded for apps that needs it. But it doesnt say which app it came from. Im sure if I mix this table with another one I could get the results. But I havent looked.

"Any authorization to add a field that already exists and is being pulled for data?"
Not sure I understand this question

Can anyone add anything, like any account add info for any JSON data and imitate an app, such as adding bogus data?
Hmm great question. I'm not a dev so I might not answer that correctly. But I am sure it requires some sort of conscious and authority, if not then it would be possible.

"Could I just add followers to people who didn't really follow? "
not unless you have the posting authority

Can I add SM data just by adding a custom JSON object? again you would need posting authority

·
·

Can anyone add anything, like any account add info for any JSON data and imitate an app, such as adding bogus data?

Yes, that's why there are lot of invalid "follow" json in the blockchain and apps should double check both the author and the content

·
·

So when a new field is created, it is associated to the original posting authority that created it, and no other account can add to that field? I thought anyone can add a follow to the custom JSON. Similarly, anyone could add any field, and fabricate data. One needs to always check the author of the data added to a field I would think.

As a non technical person I see this as a demonstration of the flexibility the blockchain has to use available resources to develop more on the ecosystem. All this activity while have the capacity to grow without impacting the spped or cost to transact! Seems like a powerful blockchain to me!

Posted using Partiko iOS

·

I'm not so technical myself. Most of the stuff I learned about the block I did so by exploring the data, I cant read code :-) so I too see this as a demonstration of the flexibility

Hi @paulag, happy new year and wish you and your family all the best, sorry it is too late to say it and celebrate it.

·

hay, how are you. Happy new year to you and your loved ones too xx

·
·

I am good, but not have more time online during this farming season, I am working in the farming land, planting veggies now. Thank you @paulag

Hi paula :) Not only that we can store custom json. We can store any type of data on the chain. Here an example where i published an image:

https://steemd.com/tx/4c6b7b2e9e242a77a6a364b19ca257e11324ce45

The data is actually shown in HEX but can be converted back to binary and then saved as a PNG file.

·

really, thats rather interesting @wehmoen, thanks for sharing :-)

Hi @paulag!

Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your UA account score is currently 7.014 which ranks you at #90 across all Steem accounts.
Your rank has not changed in the last three days.

In our last Algorithmic Curation Round, consisting of 233 contributions, your post is ranked at #2. Congratulations!

Evaluation of your UA score:
  • Your follower network is great!
  • The readers appreciate your great work!
  • Great user engagement! You rock!

Feel free to join our @steem-ua Discord server

This post has been included in the latest edition of SOS Daily News - a digest of all you need to know about the State of Steem.



Hey, @paulag!

Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!

Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).

Want to chat? Join us on Discord https://discord.gg/h52nFrV.

Vote for Utopian Witness!