Donating 5000$ and raising money with this post for covering some of SteemData costs + server upgrade. (by @Furion)
I commit to personally give 5000$ worth of STEEM or BTC at his convenience and 100% of the reward from this post and below comments to @Furion to cover server costs for hosting SteemData. Will also request from him some kind of proof of payments to verify costs.
What is SteemData?
SteemData helps developers and researchers build better STEEM applications. We parse the STEEM blockchain for you, and provide the data as a fast and convenient MongoDB service.
Here is the last update he made about SteemData
SteemData is currently available in a limited scope.
The following features are not available at the moment:
AccountOperations(Account History /w virtual ops)- Additional
AccountsData
Issues
Hardware
A full steemd node /w high-throughput is required for SteemData to function properly. It needs all plugins and all operations enabled, so that it can construct account history and store all the virtual operations alongside the operations stored on the blockchain.
Further, the node has to be in the same datacenter as SD to handle the required throughput. This is because the usage of Steem blockchain is growing, and re-syncing all the affected state requires over 100 requests per second. The private network latency within a data center is typically below 1ms, while a public network latency is usually 10 fold or more. This would decrease SD's throughput significantly.
Unfortunately, the datacenter where SteemData currently resides only offers servers with up to 256GB of RAM, and a steemd node configured to SD's requirements needs more than that.
Also, the SteemData MongoDB server is running out of disk space :(
Rusty code
I have patched things up in a quick-and-dirty fashion for a couple of months now, and the SteemData codebase has gotten a bit messy. I see this as an opportunity to clean things up, and improve reliability/performance.
Further, this is an opportunity to add infrastructure support and documentation, such that anyone can spin up their own SteemData cluster.
What needs to be done?
I am currently speccing out the new infrastructure. A new cluster will be setup in a different datacenter, capable of provisioning servers with NVMe SSD's in soft & hard RAID configs, and up to 512 GB RAM.
I should to test various steemd configurations, to achieve desired performance and provision the appropriate hardware for the next 6 months of operations.
The new DB server will also benefit from faster SSD's and larger in-RAM cache (currently 30GB, will be 60GB or 120GB).
Creating a replica set would add resilience and decouple BC processing from the database, freeing additional CPU cycles for queries. Further, adding replicas that are geo-distributed would allow for low latency in-app integrations worldwide.
As mentioned before, this is an opportunity to upgrade the operational side as well, with automated provisioning and monitoring/remediation improvements.
TL;DR: Wishlist
- Clean up the codebase
- Infrastructure as code
- Documentation
- New Servers
- Multi-Replica DB Cluster
Thank you
SteemData is a popular choice for indie developers and power users.
Steem is growing at a rapid pace, and its daily on-chain state throughput is making Bitcoin and Ethereum look pale in comparison. To scale SteemData I'm looking at the approx. recurring server costs of $3,000/mo. This would not have been possible without witness pay. Thank you for supporting my work, and thank you for using SteemData
-- @Furion
Thank You.
Contribution comment #1
Contribution comment #4
@transisto and @furion : what about data sharding the blocks chronologically and load balance from there? Of course I have no insights in steemdata's exact loads, but aren't the latest blocks queried most? If so, a relatively small chunk of data consumes most CPU / RAM but only a fraction of SSD, so by sharding "new blocks" (MongoDB can shard by default) on a separate node splits the SSD vs CPU/RAM/bandwidth problems.
Just an idea! ;-)
@scipio
EDIT: small self-upvote for visibility, 100% upvoted "contribution comment #4"
Scaling MongoDB is currently not a problem. The database would need to grow by another 1000% before sharding becomes relevant.
I think this is a very good in Amazon forest Amit user community don't forget to Upvote
Good Work Follow x Follow please <3
hola aca saludos desde venezuela me sigues te sigo :$
czechglobalhosts
Contribution comment #7
give me vote and follow i also give you vote and follow
Thank you !
Contribution comment #8
If you like the book, it is a request for reading
https://steemit.com/book/@saifuk/dan-brown-origin
Contribution comment #5
Contribution Comment
Contribution comment #6
what is this???
genau das frag ich mich auch. :-)
Auch ein deutsch-sprechender Mensch hier!
haha ich auch...
hallo! ich spreche deutsch sort of
hi.... Iam beginner in steemet follow me @rijal10 i want to share many things here and please follow me! please participate
and upvote this @sweetsssj
https://steemit.com/steem/@rijal10/how-to-register-vip-acount-bitcoin-co-id
great , follow me please
Stop asking for followers, read the post and actually contribute. Just look what happened to your rep!
Good game. Top comment. I'm so confused by this platform, but that can be a good thing...
hi.... Iam beginner in steemet follow me @rijal10 i want to share many things here and please follow me! please participate
and upvote this @sweetsssj
https://steemit.com/steem/@rijal10/how-to-register-vip-acount-bitcoin-co-id
Did you even read my comment?
Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me
Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me
Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me
SteemData looks interesting might check it out on some future project. Not been 100% sure of current APIs.
@transisto and @furion So many great developments on the steem blockchain, and this is a great one to get more smart and creative developers on the block. I just upvoted both the post, and this comment.
By the way @transisto , I made a reply in your previous post of the size of a normal post, and your feedback is very important to me.
Here is a link to your previous post and my comment.
https://steemit.com/steem/@transisto/my-version-of-steem-is-not-content-based-it-is-contribution-based#@gold84/re-transisto-my-version-of-steem-is-not-content-based-it-is-contribution-based-20171230t142131356z
Looking forward to hear from you!
Regards, @gold84
yeah man there is so much demand for new deve;opers and great upvpyte potential for them that we should make a special front end just to post steem development projects and it will have a list of supporters, peopel who delegate steempower or curate themselves to peopel who post godo proejcts, kinda like @utopian-io for dvelopers, it would be really nice, it can allow steem to have a WHOLE github like system with slack as well, we should REALLY make the STEEM BROWSER and have a whole Steem Giuthub type website where tseem posts temselves hold the code, and just replace github repositories with steemit pots and comments
@ackza I totally agree with you! Thanks for adding this great idea to the message. Hope to hear from you more oftern on my post ideas/chapters. You have a lot to share there.
I am not sure if you know I am completing a full 11 posts/chapters steemit guide for new visitors, new steemit users, minnows, for them to learn as much as possible and as easier they can about steemit and the steem blockchain, to make their way here. In my already first 2 chapters published I got lots of questions, comments, ideas, and I believe it is helping them.
Looking forward to your thoughts when you get a chance.
Regards, @gold84
with this guys supporting steemit, steemit can only get huge!! we believe it
What is this contribution thing ? Guys
Explain me please
He wrote that all post rewards goes to funding the project including rewards from comments . So if I want to support them with more than 1 full upvote, I can both upvote his post and one or more of his comments. That's why.
Next time I'll stack them all like this so no random comment can fuck up the order.
Explanation for high upvote
Even if you stack them like this, this does not guarantee that others will not comment and break the order of your comments. And I guess the last comments won't be visible in the same page so there are sone disadvantages...
#1
#2 ...
Good innovation upvoted
contribution comment #1.5
contribution comment #1.55
great , follow me please
Okie lets do it...
My Contribution Comment :)
please up vote my posts as u have strong steemit power
Bro, how are you able to get this huge upvote. I am new on steem can you please show me the way?
Just for upvotes you can use @booster bot
How do I know it worked. Is it @boosterbot
Or
@booster bot
If you like the book, it is a request for reading
https://steemit.com/book/@saifuk/dan-brown-origin
Using the same comments that has nothing to do with the post over and over again may give you some reports. I can see that you mostly write good comments, so I wont report you this time, but please dont spam
Contribution comment #21
Off😂
@transisto It's such a great pleasure to be part of this contribution. I strongly believe the turnup is gonna be massive.
@tomsnoob
Contribution, contribution... Let`s go working
my last post contains a free gift Happy New year everyone!
Thanks...!
great
Comentario de contribución desde Venezuela, Excelente (Y)
I didn't understood what is the work but still if you agree to do something good for steemit community .. this is worth commenting 👍
My last post contains a gift, Happy New year everyone!
Great job.
give me vote and follow i also give you vote and follow
voted and followed
Dont follow people that ask for followers without contributing to anything. He wont be a loyal follower, and he is spamming. The comment is copy paste
Contribution comment #22
give me vote and follow i also give you vote and follow
Contribution comment #2
Contribution Comment #2
I love you !
Contribution comment #2
Keep up the good works..power!
Dear friends, as my friend had taken the world steemit be happy
contribution comment #2.5
Contribution comment #3
contribution comment #3.5
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Maybe this has been asked somewhere already...
Why does Furion actually need more help with server costs? He has been one of the top witnesses for a while now and as one in the top-20, he is earning 7000-8000 SP per month. At current prices (~$2.85 right now), that's $20,000+ per month vs. ~$3000/mo. renting the equipment. Break-even on the new projected costs would be a $0.50 STEEM price.
For a few thousand dollars, you can buy your own 512 GB RAM server. Cash out a few thousand STEEM from witness rewards just from this month and he can buy two.
Not too long ago, Steem was at 80c.
You're right, I should just power down some Steem to cover the costs. I do however feel bad whenever I do that.
SteemData is well funded given the recent price increase and my witness position. I would like to forward the funds raised in this campaign to other Steem projects.
There’s no need to feel bad for powering down, especially since the witness rewards are received as SP. It’s the only option.
And just to clarify - I get the costs of your project. I just didn’t understand why ~$1000 of post rewards were needed when witness rewards are 20+ times more than that...per month (at current prices). If you cashed some out tonight, you could potentially get about $5 per STEEM. Seems like a great deal to me!
Good luck with your new servers and upgrades! I have a feeling that most of us will be upgrading witness specs again soon with STEEM prices continuing upward.
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
That's why I think, that steem does not scale.
512GB Node ..that is crazy. No one, except rich or already SP rich witnesses could afford running this
STEEM goes further and further away from decentralization. heading into a 20 Witness "diversivied" centralized system
Steemdata is a backend that powers down stream applications. It will have processing and throughput needs over and above that needed to maintain a blockchain node. I’ve used the api and written big queries... it needs an upgrade
like getting a good post
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
I know..I am good!
Chers!
Hasn't he already crowd-funded some equipment before/
I’m not sure about equipment, but there was quite a few thousand allocated via posting rewards for SteemQ, which then turned into Viewly. And last I saw (3 months ago), there was to be an ICO for Viewly. Haven’t seen or heard anything about it since then. Furion’s social activity on the blockchain has only been one comment in the past two months...and that comment was worth $190+ when I saw it earlier today.
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Dear friend
Like getting a good graphics
I announced you
you are a The world famous grapher
Dear friend
Like getting a good graphics
I announced you
you are a The world famous grapher
wow excellent post
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
SteemData is really great and I am using it for some features for dMania.
Having more options to query the Steem blockchain makes a lot more features and apps on Steem possible. That is why a service like this is very important.
Lately it was lagging behind and the data wasn't up to date sometimes. I guess it was because of the limited hardware resources.
If SteemData is reliable and all the data is up to date, I think it will become crucial for the growth of Steem and we will see a lot of new great apps and features.
@Furion don't forget to add more indices for comments 😅 I really need them.
I agree with you we cultivate
Hi there @zombee I have been trying to post more on dmania so does that mean like I m doing something wrong ? I feel I can make people laugh. So I try to make more post ... Then today I realised not many people are looking into it .. n my post promotions went worthless ... So I m not sure should j make more post or just a few
dMania is still growing. Every post and contribution helps us grow. Thanks for your support.
& truth be told I m having zero knowledge of coding .. I can barely understand what to do with these kind of post ... But I try to searc keywords mentioned in these .. n m reading everything line 12 hours a day
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Contribution comment
This is some good stuff. I'm glad to hear that we are improving the infrastructure used by the tools used by the community.
Awesome. Voted!
interesting! I want to use SteeemData later
เห็นด้วยอย่างยิ่ง @nongboy
https://steemit.com/@nongboy
Once I want to get a fund for the 'Rohingya' refeuges in steemit ..... But I cant .. I am not big enough to do that.. Also I am not big enough on steem as well ... So I cancel the dream to raise the fund and help them what I can effort. But you doing a great job big brother. Help the people .. Maybe Allah help you. Insha Allah you get success in this work..