Exploratory Analysis Shows World of Opportunity with JSON and STEEM

in #utopian-io5 years ago

The use of Custom JSON field on the STEEM Blockchain is becoming more and more popular.  From discussion I have read, this field can do almost anything from creating Tokens to holding game information.

JSON data is a way of representing objects or arrays.  It is easy to read, and it is easy to parse.  Many API calls will return JSON format and many web apps use JSON to easily move information around the internet.

Its is therefore no surprise that developers are now using the JSON filed on steem to move data between an app and the blockchain.

Initially the Custom JSON field was used for holding follow information, where one account on the blockchain follows another.  However, over time, this has expanded to holding information from multiple sources for multiple reasons.

With that in mind I wanted to take a quick look at the Txcustoms table in the STEEMSQL database to just get an idea of what and who is using this JSON field.  This type of analysis is commonly called exploratory analysis.

Repository

https://github.com/steemit/steem

Aim of Exploratory Analysis

- Get a feel for the Txcustoms table in STEEMSQL.

- Establish if there was growth in the use of this field from 2018 to 2019

- Establish what app is using this most

- Get an idea of the type of data held 

Analysis

The table below shows the custom JSONs for the first week of January 2018. None of the Custom JSONs required additional posting or other authorities.  As we can see, most of the transitions using this field were Follows, however there were a few others too.

 

Pacman-live seems to be one of the first games to host data on the blockchain.  Looking at this further the data sent to the blockchain was the username, the score, the game level and a timestamp.

  

Chainbb seem to have been using the JSON field to store details of Forum objects, such as forum configurations and forum posts.

  

Distribution is an interesting one and seems to be holding information about distributions from randowhale

  

 And we can also see some testing of STEEMMonsters under TEST_sm_pre_sale_packs

  

Jumping on now to the first week of January 2019 and things are looking rather different. 980162 custom JSON fields were used, down 58% on the same time last year.  However, this is not bad news because in 2018 99.8% of these related to follows and unfollows.  With the downturn in activity on the blockchain, for the same period in 2019 follows and unfollows make up only 9.8% of the custom JSON transactions.  The was a reduction of 95% in the actual number of follows and unfollows.

  

The table above shows the number of transitions of each time with custom JSONS. I have split this list in two.  On the right we have all the STEEMMonsters custom JSONS.  This makes up 88% of all custom JSONs.  On the left we have others such as Follows, DiceHash and Dlux.

The table below shows a sample of the data contained within some of the JSONs.

 

Also changed considerably since 2018 is the user of authority to post and this is also recorded in the blockchain and the SQL database.  Looking at this for STEEMMonsters in the table below, by doing a count of the required_posting_auths we can see that for this period there were 366,192 find match and by carrying out a distinct count of the same field, we can establish this was by 2379 different accounts.

  

Conclusion

Although there was a reduction in the number of custom JSONs used from Jan 18 to Jan 19, this is a reflection on the downturn in social activity on the block and not development activity.

The social activity, follows and unfollows is down 95%.  However, in 2018 non-social activity recorded in JSON was only 0.02%, in 2018 this was 90.2%.  This is an indication of development on the block and a sign in the shift of the platform from being a content/social platform to something more.

By looking at the data contained within the JSONs my eyes have been opened on how simple it can be to really make ‘ANY’ app a DApp on the STEEM Blockchain and I hope it gives you that sense too.  

Obviously STEEMMonsters are well out there in terms of usage compared to other apps using JSONs but many apps are only being to test this feature.  Some of this comes from the shift away from Steemit.Inc as the only developers on the block.  STEEM is coded in C++ which is a skill many do not have.  However, there is no shortage of App developers with JSON knowledge.

Now that I have a better understanding of how this table in the SQL database is kept and what data it contains, it opens up a new world of analysis.

So, who will be first to have a D+ site ready to replace G+ communities using our new friend JSON?

Data and Query

The data for this post was taken from the STEEMSQL database held and managed by @arcange.

The query used for 2019 was

select *
from Txcustoms
where CONVERT(DATE,timestamp) BETWEEN '2019-01-01' AND '2019-01-07'

and for 2018 was

select *
from Txcustoms
where CONVERT(DATE,timestamp) BETWEEN '2018-01-01' AND '2018-01-07'

As always, I used PowerBI to

- Connect to the data

- Transform the data

- Carry out calculations

- Visualise the data

If you want to learn more about using PowerBI or Excel do follow @theexcelclub


Sort:  

Hi @paulag, great topic, the custom-JSONs can be used for a number of things, and have seen quite some popularity with several apps and projects on top of Steem. Steemmonsters and MagicDice are probably the most prominent use case as of now. Unfortunately, SteemSQL is not fully ready for analyzing this yet, so the senders of the ops are only partly available at the moment. I would have loved to see more data covered than two periods of one week each. I could imagine having maybe weekly counts for a couple of tids could have provided additional insights here?

Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.

To view those questions and the relevant answers related to your post, click here.


Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]

Thank you for your review, @crokkon! Keep up the good work!

Cool.
Is there a record of who adds the custom JSON?
Any authorization to add a field that already exists and is being pulled for data?
Can anyone add anything, like any account add info for any JSON data and imitate an app, such as adding bogus data?
Could I just add followers to people who didn't really follow?
Can I add SM data just by adding a custom JSON object?

Thanks!

Great questions

"Is there a record of who adds the custom JSON?"
In that table, not really, but there is posting auth recorded for apps that needs it. But it doesnt say which app it came from. Im sure if I mix this table with another one I could get the results. But I havent looked.

"Any authorization to add a field that already exists and is being pulled for data?"
Not sure I understand this question

Can anyone add anything, like any account add info for any JSON data and imitate an app, such as adding bogus data?
Hmm great question. I'm not a dev so I might not answer that correctly. But I am sure it requires some sort of conscious and authority, if not then it would be possible.

"Could I just add followers to people who didn't really follow? "
not unless you have the posting authority

Can I add SM data just by adding a custom JSON object? again you would need posting authority

So when a new field is created, it is associated to the original posting authority that created it, and no other account can add to that field? I thought anyone can add a follow to the custom JSON. Similarly, anyone could add any field, and fabricate data. One needs to always check the author of the data added to a field I would think.

As a non technical person I see this as a demonstration of the flexibility the blockchain has to use available resources to develop more on the ecosystem. All this activity while have the capacity to grow without impacting the spped or cost to transact! Seems like a powerful blockchain to me!

Posted using Partiko iOS

I'm not so technical myself. Most of the stuff I learned about the block I did so by exploring the data, I cant read code :-) so I too see this as a demonstration of the flexibility

Hi @paulag, happy new year and wish you and your family all the best, sorry it is too late to say it and celebrate it.

hay, how are you. Happy new year to you and your loved ones too xx

I am good, but not have more time online during this farming season, I am working in the farming land, planting veggies now. Thank you @paulag

really, thats rather interesting @wehmoen, thanks for sharing :-)

Hi @paulag!

Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your UA account score is currently 7.014 which ranks you at #90 across all Steem accounts.
Your rank has not changed in the last three days.

In our last Algorithmic Curation Round, consisting of 233 contributions, your post is ranked at #2. Congratulations!

Evaluation of your UA score:
  • Your follower network is great!
  • The readers appreciate your great work!
  • Great user engagement! You rock!

Feel free to join our @steem-ua Discord server

This post has been included in the latest edition of SOS Daily News - a digest of all you need to know about the State of Steem.



Hey, @paulag!

Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!

Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).

Want to chat? Join us on Discord https://discord.gg/h52nFrV.

Vote for Utopian Witness!

Coin Marketplace

STEEM 0.35
TRX 0.12
JST 0.040
BTC 70351.33
ETH 3563.43
USDT 1.00
SBD 4.72