Roadmap for SteemData 2.0 ∙ Crowdfunding
Last week I launched a preview of SteemData, and the response, both public and private has been fantastic.
Based on your feedback, I've created a roadmap for SteemData 2.0.
Main Features
Real-Time
One of the major drawbacks of the current design is that the database is lagging behind blockchain state. While this is sufficient for a large array of applications, it is not optimal for certain kinds of user facing apps that rely on fresh data.
A major re-design to an event based model is required, so that when new blocks become available, all relevant parts of the database get updated in near-real-time.
Native Data Types
A vast majority of the data in the current version is under-typed. This makes it hard for people to make good queries, and often requires additional post-processing to get the results we're interested in. SteemData 2.0 needs to address these issues.
Indexes
The indexes need to be studied and optimized to fit real world usage patterns. Furthermore, it would be nice to have relationships between objects in different collections.
Backups
It takes more than a week to re-create (sync) the database from scratch. This is why SteemData needs to implement automatic Amazon S3 snapshots, so that in the event of catastrophic failure, the service can be brought back to life with minimum downtime.
Open Source
SteemData is written in Python and uses Docker services for deployment/orchestration.
The software stack needs to be refactored, documented and made available so that anyone can deploy their own version of SteemData for personal use.
Furthermore, reference implementations for Python and JavaScript clients as well as helpers utilities should be created.
The license will be MIT, as it is highly permissive and basically grants users the power to do whatever they like. This will hopefully also help developers create support for different databases (SQL, Firebase, etc).
More Features
- Proper Integration of Post Comments
- Restructure Operations
- Support for HF 16.1 and beyond
- Historic Price Feeds
- Steemle like charts on SteemData.com
Target shipping date: March 1st 2017
Donations
Steem community has proven that a donation model can work as Busy raised close to $60,000 for the awesome work they're doing. This is awesome.
The main goal for SteemData is help developers create new and interesting applications for the STEEM ecosystem. I believe that the ecosystem will play an important role in Steem's future success, and I would love to contribute in my humble little way.
Crowdfunding
To fund the development of SteemData 2.0 I am looking to raise $5,000 in STEEM or SBD.
The donations should be sent to @steemdata, and the list of friendly donors will be published and updated here, as well as in future announcements.
Jan 22nd: We have raised $2,600 of the $5,000 goal so far.
| Supporters | |
|---|---|
| @cass | $2,500 |
| @fabien | $100 |
I understand you @furion. It can be challenging getting project off the ground even when people show interest. Actions are needed to be taken and sometimes it does not depend solely on dvlpmt. SteemQ is a huge project which has to have a lot support and dedication. Kind of busy. SteemData is probably needed now by community dev, so to build other tools, compare to SteemQ which is mostly mainstream product. I support SteemData crowdfunding.
I will contribute $2.5k !
Looking great, brother! When do you think it will be opensource? I think it can be very useful to have quick start and replicability of steemdata. Looking forward for this as I think there are great opportunities on few projects with steemdata.
Great news all together really! Thanks for sharing, good luck for our future and namaste :)
what about money (4k SBD) which you got from post about SteemQ? I didn't see any updates about its development
I have had trouble getting traction on SteemQ, and thus the project has been shelved. There is a lot of interest in the idea, but I've mostly hit rejections and limitations that I could not overcome.
what if you won't get traction with steemdata-2-0 also? Maybe it is better idea, to start development and give frequent updates at the same time. Then people would know what they actually receiving.
People are still waiting for SteemQ. 557 people voted for your post about SteemData and 815 for post about SteemQ. It is clear, which project has greater potential.
SteemData is already functional and out there. Check it out https://steemdata.com
I am raising a crowdfund to make it a lot better, and I have the skills necessary to get it done.
of course there's traction, steemQ was linked all over the place. the only complaints about it are that there are no updates so must've been some sort of trick or it is too hard to do but that's not what you said. One way or the other something like it will happen for any data, making copyright enforcement a thing of our dark past.
Might also be a timing thing ... the supporters might have been temporarily broke and it might take a while for others who would be supporters to find out about it.
SteemData ... or something forked from it or inspired by it ... will eventually get traction. The development effort is appreciated!
This is my following dot, so I can easily find this post again via my comments section.
@whatsup I've tried doing the comment thing to keep track of posts, right now i'm embedding JSON in transactions, see my post where I extract the transfers from steemdata using node.js. I plan on creating web page to display bookmarks. send to @msteem. Replace link field with your bookmark.
{"type":"bookmark","action":"add","version":"00","link":"https://steemit.com/steemdata/@furion/roadmap-for-steemdata-2-0-help-needed","comment":"nice to have"}
@furion really been trying to get in touch. Is there an easier way to reach out to you? I'm looking at steemdata to be a central part of my business. Would like to discuss business and just bullshit. Concerning updates in real-time, you might want to look at ZeroMQ.
I've been planning to use RabbitMQ with Celery. I will definitely look into ZeroMQ as well.
Just in case anyone is not familiar with this detailed comparison ... ZeroMQ is like Git (essentially distributed) ... whereas RabbitMQ [one implementation of AMQP; OpenAMQ is another] is like SVN (essentially centralized).
Of course, there can certainly be very good, very practical reasons for using a centralized framework ... probably depends mostly on the audience of people involved ... it's like buying a sandwich from a vendor who's only set up to take coins, dollar bills and maybe some credit cards, ie trying to pay with bitcoin means no sandwich or something even worse such as the cops being called.
You might also want to look at the forks of ZeroMQ ... as well as CurveZMQ, protocol for secure messaging for ZeroMQ applications across the Internet which uses Daniel J. Bernstein's CurveCP security handshake as an encryption layer.
Especially noteworthy is the fork by Martin Sustrik (the main original developer responsible for the code behind ZeroMQ) who has started nanomsg as a POSIX-compliant rewrite of the ZeroMQ core library in C, rather than C++.