Streaming the blockchain and ingestion (Free Knowledge sharing 0.1 )
What is this ?
I am trying probably something new -- sharing initial versions of the documents / notes on the projects I am working on. The reasoning is,
- sometimes the ideas are good enough to be shared
- Many times we need inputs from people who have already done it to make the design better
- Obviously it will also act like a advertisement to attract quality people :-)
So this series of "thoughts" are open to criticisms, suggestions, questions and Criticisms are more important!
Streaming the blockchain and ingestion
To provide the middle-ware its important that we have a near-real time copy of the data from the blockchain in a database. This feature will also provide the foundations for establishing a snap shot of the database and thus reducing the RAM needs.
The conceptual architecture
From the blockchain we will streaming the data using the RPC (gRPC etc can be considered later). As the first step this will be stored in a in-memory cache (Redis) for the further processing. This in-memory caching is used to provide real time capture of data and if required expose the data to other "north bound" DApps in "near real time".
[]-[]-[]-[]-[] ------> * (stream logic) -----> mDB
In the step2, in-memory DB data will be stored to MongoDB.
We will consider the blockchain as the producer and the streaming logic as the consumer
create two thred pools one consuming the data from the genesis block and the other consuming the data from the last irriversible block in the chain
The thread / worker pools can be implemented in Java or in Python (though we will have to fight the GIL limitations)
Since the data is not going to be the order to Tera bytes like in a typical Big data scenario, we will not be using a Message Queue / AMQT & considering this in a later stage.
The project also has wallet, historical information etc and these will be taken from the micro services APIs exposed using the streaming + ingestion component
real time data from the blockchain is not the scope of this project
In the context of STEEM blockchain SBDS and chainsync are two interesting projects with similar goals
STEEM blockchain also has many MongoBD based (third party) data providers
Databases
For now we will be proceeding with MongoDB. In the future multiple sinks like Elastic Search or RethinkDB will be selected based on the requirements
Caching
We will soon have to provide in memory caching and as of now Redis is selected for the purpose.
assuming that you are "Streaming" the blockchain
0-get-block-num
1-store-in-redis/memcache
2-proceed-with-application logic
3-.app restarts, crashes
4- fetch the blocknumber from redis
5.-compare with last compared block in the app
get-next bloc
else
start with get-block-from-redis
This post has been rewarded with 30% upvote from @indiaunited-bot account. We are happy to have you as one of the valuable member of the community.
If you would like to delegate to @IndiaUnited you can do so by clicking on the following links: 5SP, 10SP, 15SP, 20SP 25SP, 50SP, 100SP, 250SP. Be sure to leave at least 50SP undelegated on your account.
to get community support and guidance
. Please contribute to the community by upvoting this comment and posts made by @indiaunited.
This post has been rewarded with 100% upvote from @indiaunited account as part of Curation program.
perfect dear!!
Hey @bobinson! you seem like a steemit power user. Given you're one of my followers, any chance you'd like to try our new cross platform mobile app? We've added things missing from steemit like push notifications, user mentions, and chatting with other users. We’re also giving away free STEEM if you invite new users to the app. You can download the app here for iOS:
https://itunes.apple.com/us/app/vapor-a-steem-app/id1359525246
or here for Android:
https://play.google.com/store/apps/details?id=com.vapor
We're always looking ways to improve the app, so if you have any feedback we'd love to hear it!
successful friends, let's exchange vout by way of mutual visit to our blog each @zuhrafriska