Update on web scalable EOS History API
EOS Tribe have implemented an additional History API endpoint for /v2/history/get_key_accounts
Below is our logical diagram for ElasticSearch cluster supporting currently 3 ES nodes with data shared across them:
All History API source code for Elasticsearch is available at our public repo:
https://github.com/EOSTribe/ESHistoryAPI
Special thanks to EOSLaoMao for developing Elasticsearch plugin used:
https://github.com/EOSLaoMao/elasticsearch_plugin
As of a time of this article on December 23, 2018 at 7:30am Los Angeles time our Elasticsearch cluster is synced up to 27931400 blocks and all historical data is available up to this block.
We will make a separate announcement once our Elasticsearch cluster is fully synced with the Blockchain.
Meanwhile we are working on remaining History endpoint.
All matching /v1/history/* endpoints are now redirected to available /v2/history/* endpoints and both available at https://api.eostribe.io/v2/history/*
EOS Tribe updated Infrastructure diagram:
I just resteemed your post!
Why? @eosbpnews aggregates updates of active EOS BPs and conveniently serves them in one place!
This service is provided by @eosoceania. If you think we are doing useful work, consider supporting us with a vote :)
For any inquiries/issues please reach out on Telegram or Discord.
Nice stuff... really like where this is going. =)
I wonder how big is that elasticsearch cluster. At my work, I have only the minimum 3 node cluster, and I would love to play more with it if I could. Since in HPC we work mainly on Infiniband networks (behind the scenes). Which is where I would see great performance for scalable elastic search clusters backends. Most of the communication between them is basically latency bound (something you can't yet do on AWS).
Cheers
I have to get back to you on ES cluster size after we are fully synced.
We only record data in ES that is used by History API. It saves a lot of space.
Yep, I agree, and besides saving space, is very scalable (machine wise)! And that's the beauty of ES.
Also, because History has lots of "ends" in my view and can be dealt with in several ways. I can only see several standards emerging, and the use of a single API format for consumers will gradually disappear (my view). But I like the idea from @greymass (I believe it was from them) saying the API should be queriable for the format, after being proposed and accepted onchain. Then anyone can talk to the format they most wish... and even select the "type" of data (if the format allows you so) you want to access.
And from here, I see lots of advantages in having the history data in several types of databases. Some might even have offline data that if needed to be accessed, can be, but at a cost (of time). It would also allow many savings on "hot" data since this one can be stored/accessed in many IO performance flavours. Not necessarily has to be in the API, but would help if the API had some kind of tiering that eases the user cost (in case speed is not needed) or provides performance requirements if the user is keen on paying for it.
Anyway, I would really like to see the ES solution more adopted, to be honest. Along with some of the other tweaks I have been seeing around where to go with the History API format.
Cheers and Happy New Year!
We are at 31M+ blocks out of 34M blocks as of Dec 26, 2018.
Hopefully we sync up to Head block before the New Year ;-)