Donating 5000$ and raising money with this post for covering some of SteemData costs + server upgrade. (by @Furion)

in #steem3 years ago (edited)

I commit to personally give 5000$ worth of STEEM or BTC at his convenience and 100% of the reward from this post and below comments to @Furion to cover server costs for hosting SteemData. Will also request from him some kind of proof of payments to verify costs.

What is SteemData?

SteemData helps developers and researchers build better STEEM applications. We parse the STEEM blockchain for you, and provide the data as a fast and convenient MongoDB service.

https://steemdata.com/

Here is the last update he made about SteemData




SteemData is currently available in a limited scope.

The following features are not available at the moment:

  • AccountOperations (Account History /w virtual ops)
  • Additional Accounts Data

Issues

Hardware

A full steemd node /w high-throughput is required for SteemData to function properly. It needs all plugins and all operations enabled, so that it can construct account history and store all the virtual operations alongside the operations stored on the blockchain.

Further, the node has to be in the same datacenter as SD to handle the required throughput. This is because the usage of Steem blockchain is growing, and re-syncing all the affected state requires over 100 requests per second. The private network latency within a data center is typically below 1ms, while a public network latency is usually 10 fold or more. This would decrease SD's throughput significantly.

Unfortunately, the datacenter where SteemData currently resides only offers servers with up to 256GB of RAM, and a steemd node configured to SD's requirements needs more than that.

Also, the SteemData MongoDB server is running out of disk space :(

Rusty code

I have patched things up in a quick-and-dirty fashion for a couple of months now, and the SteemData codebase has gotten a bit messy. I see this as an opportunity to clean things up, and improve reliability/performance.

Further, this is an opportunity to add infrastructure support and documentation, such that anyone can spin up their own SteemData cluster.

What needs to be done?

I am currently speccing out the new infrastructure. A new cluster will be setup in a different datacenter, capable of provisioning servers with NVMe SSD's in soft & hard RAID configs, and up to 512 GB RAM.

I should to test various steemd configurations, to achieve desired performance and provision the appropriate hardware for the next 6 months of operations.

The new DB server will also benefit from faster SSD's and larger in-RAM cache (currently 30GB, will be 60GB or 120GB).

Creating a replica set would add resilience and decouple BC processing from the database, freeing additional CPU cycles for queries. Further, adding replicas that are geo-distributed would allow for low latency in-app integrations worldwide.

As mentioned before, this is an opportunity to upgrade the operational side as well, with automated provisioning and monitoring/remediation improvements.

TL;DR: Wishlist

  • Clean up the codebase
  • Infrastructure as code
  • Documentation
  • New Servers
  • Multi-Replica DB Cluster

Thank you

SteemData is a popular choice for indie developers and power users.

Steem is growing at a rapid pace, and its daily on-chain state throughput is making Bitcoin and Ethereum look pale in comparison. To scale SteemData I'm looking at the approx. recurring server costs of $3,000/mo. This would not have been possible without witness pay. Thank you for supporting my work, and thank you for using SteemData

-- @Furion





Thank You.

Sort:  

@transisto and @furion : what about data sharding the blocks chronologically and load balance from there? Of course I have no insights in steemdata's exact loads, but aren't the latest blocks queried most? If so, a relatively small chunk of data consumes most CPU / RAM but only a fraction of SSD, so by sharding "new blocks" (MongoDB can shard by default) on a separate node splits the SSD vs CPU/RAM/bandwidth problems.

Just an idea! ;-)
@scipio

EDIT: small self-upvote for visibility, 100% upvoted "contribution comment #4"

Scaling MongoDB is currently not a problem. The database would need to grow by another 1000% before sharding becomes relevant.

I think this is a very good in Amazon forest Amit user community don't forget to Upvote

Good Work Follow x Follow please <3

hola aca saludos desde venezuela me sigues te sigo :$

give me vote and follow i also give you vote and follow

genau das frag ich mich auch. :-)

Auch ein deutsch-sprechender Mensch hier!

haha ich auch...

hallo! ich spreche deutsch sort of

hi.... Iam beginner in steemet follow me @rijal10 i want to share many things here and please follow me! please participate

and upvote this @sweetsssj
https://steemit.com/steem/@rijal10/how-to-register-vip-acount-bitcoin-co-id

Stop asking for followers, read the post and actually contribute. Just look what happened to your rep!

Good game. Top comment. I'm so confused by this platform, but that can be a good thing...

hi.... Iam beginner in steemet follow me @rijal10 i want to share many things here and please follow me! please participate

and upvote this @sweetsssj
https://steemit.com/steem/@rijal10/how-to-register-vip-acount-bitcoin-co-id

Did you even read my comment?

Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me

Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me

Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me

SteemData looks interesting might check it out on some future project. Not been 100% sure of current APIs.

@transisto and @furion So many great developments on the steem blockchain, and this is a great one to get more smart and creative developers on the block. I just upvoted both the post, and this comment.

By the way @transisto , I made a reply in your previous post of the size of a normal post, and your feedback is very important to me.

Here is a link to your previous post and my comment.
https://steemit.com/steem/@transisto/my-version-of-steem-is-not-content-based-it-is-contribution-based#@gold84/re-transisto-my-version-of-steem-is-not-content-based-it-is-contribution-based-20171230t142131356z

Looking forward to hear from you!

Regards, @gold84

yeah man there is so much demand for new deve;opers and great upvpyte potential for them that we should make a special front end just to post steem development projects and it will have a list of supporters, peopel who delegate steempower or curate themselves to peopel who post godo proejcts, kinda like @utopian-io for dvelopers, it would be really nice, it can allow steem to have a WHOLE github like system with slack as well, we should REALLY make the STEEM BROWSER and have a whole Steem Giuthub type website where tseem posts temselves hold the code, and just replace github repositories with steemit pots and comments

@ackza I totally agree with you! Thanks for adding this great idea to the message. Hope to hear from you more oftern on my post ideas/chapters. You have a lot to share there.

I am not sure if you know I am completing a full 11 posts/chapters steemit guide for new visitors, new steemit users, minnows, for them to learn as much as possible and as easier they can about steemit and the steem blockchain, to make their way here. In my already first 2 chapters published I got lots of questions, comments, ideas, and I believe it is helping them.

Looking forward to your thoughts when you get a chance.

Regards, @gold84

with this guys supporting steemit, steemit can only get huge!! we believe it

What is this contribution thing ? Guys

Explain me please

He wrote that all post rewards goes to funding the project including rewards from comments . So if I want to support them with more than 1 full upvote, I can both upvote his post and one or more of his comments. That's why.

Next time I'll stack them all like this so no random comment can fuck up the order.

Even if you stack them like this, this does not guarantee that others will not comment and break the order of your comments. And I guess the last comments won't be visible in the same page so there are sone disadvantages...

#1

#2 ...

DQmVXKKuHaYTK3VMKvswmon4p3ZcSv1iP3BY7Vd5kuB9toh.png

Good innovation upvoted

contribution comment #1.5

Okie lets do it...
My Contribution Comment :)

please up vote my posts as u have strong steemit power

Bro, how are you able to get this huge upvote. I am new on steem can you please show me the way?

Just for upvotes you can use @booster bot

How do I know it worked. Is it @boosterbot
Or
@booster bot

If you like the book, it is a request for reading
https://steemit.com/book/@saifuk/dan-brown-origin

Using the same comments that has nothing to do with the post over and over again may give you some reports. I can see that you mostly write good comments, so I wont report you this time, but please dont spam

Contribution comment #21

@transisto It's such a great pleasure to be part of this contribution. I strongly believe the turnup is gonna be massive.

@tomsnoob

Contribution, contribution... Let`s go working

my last post contains a free gift Happy New year everyone!

Comentario de contribución desde Venezuela, Excelente (Y)

I didn't understood what is the work but still if you agree to do something good for steemit community .. this is worth commenting 👍

My last post contains a gift, Happy New year everyone!

give me vote and follow i also give you vote and follow

voted and followed

Dont follow people that ask for followers without contributing to anything. He wont be a loyal follower, and he is spamming. The comment is copy paste

give me vote and follow i also give you vote and follow

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Maybe this has been asked somewhere already...

Why does Furion actually need more help with server costs? He has been one of the top witnesses for a while now and as one in the top-20, he is earning 7000-8000 SP per month. At current prices (~$2.85 right now), that's $20,000+ per month vs. ~$3000/mo. renting the equipment. Break-even on the new projected costs would be a $0.50 STEEM price.

For a few thousand dollars, you can buy your own 512 GB RAM server. Cash out a few thousand STEEM from witness rewards just from this month and he can buy two.

Not too long ago, Steem was at 80c.

You're right, I should just power down some Steem to cover the costs. I do however feel bad whenever I do that.

SteemData is well funded given the recent price increase and my witness position. I would like to forward the funds raised in this campaign to other Steem projects.

There’s no need to feel bad for powering down, especially since the witness rewards are received as SP. It’s the only option.

And just to clarify - I get the costs of your project. I just didn’t understand why ~$1000 of post rewards were needed when witness rewards are 20+ times more than that...per month (at current prices). If you cashed some out tonight, you could potentially get about $5 per STEEM. Seems like a great deal to me!

Good luck with your new servers and upgrades! I have a feeling that most of us will be upgrading witness specs again soon with STEEM prices continuing upward.

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

That's why I think, that steem does not scale.
512GB Node ..that is crazy. No one, except rich or already SP rich witnesses could afford running this
STEEM goes further and further away from decentralization. heading into a 20 Witness "diversivied" centralized system

Steemdata is a backend that powers down stream applications. It will have processing and throughput needs over and above that needed to maintain a blockchain node. I’ve used the api and written big queries... it needs an upgrade

like getting a good post

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

I know..I am good!
Chers!

Hasn't he already crowd-funded some equipment before/

I’m not sure about equipment, but there was quite a few thousand allocated via posting rewards for SteemQ, which then turned into Viewly. And last I saw (3 months ago), there was to be an ICO for Viewly. Haven’t seen or heard anything about it since then. Furion’s social activity on the blockchain has only been one comment in the past two months...and that comment was worth $190+ when I saw it earlier today.

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Dear friend
Like getting a good graphics
I announced you
you are a The world famous grapher

Dear friend
Like getting a good graphics
I announced you
you are a The world famous grapher

wow excellent post

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

SteemData is really great and I am using it for some features for dMania.
Having more options to query the Steem blockchain makes a lot more features and apps on Steem possible. That is why a service like this is very important.

Lately it was lagging behind and the data wasn't up to date sometimes. I guess it was because of the limited hardware resources.
If SteemData is reliable and all the data is up to date, I think it will become crucial for the growth of Steem and we will see a lot of new great apps and features.

@Furion don't forget to add more indices for comments 😅 I really need them.

I agree with you we cultivate

Hi there @zombee I have been trying to post more on dmania so does that mean like I m doing something wrong ? I feel I can make people laugh. So I try to make more post ... Then today I realised not many people are looking into it .. n my post promotions went worthless ... So I m not sure should j make more post or just a few

dMania is still growing. Every post and contribution helps us grow. Thanks for your support.

& truth be told I m having zero knowledge of coding .. I can barely understand what to do with these kind of post ... But I try to searc keywords mentioned in these .. n m reading everything line 12 hours a day

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Contribution comment

This is some good stuff. I'm glad to hear that we are improving the infrastructure used by the tools used by the community.

Awesome. Voted!

interesting! I want to use SteeemData later

เห็นด้วยอย่างยิ่ง @nongboy
https://steemit.com/@nongboy

Once I want to get a fund for the 'Rohingya' refeuges in steemit ..... But I cant .. I am not big enough to do that.. Also I am not big enough on steem as well ... So I cancel the dream to raise the fund and help them what I can effort. But you doing a great job big brother. Help the people .. Maybe Allah help you. Insha Allah you get success in this work..

Data Maintenance and Server upgrade it is really costly. I appreciate about the work. Really it is talented work.

[Prediction] The next U.S. Presidential election will use blockchain technology and it will be totally transparent and fraud-proof!

Ok guys get start being new year dhamaka happy new year 2018 in advance

I always miss out on the contribution comments :(

That is a very good cause for raising money!
Apparently, Steem rewards can go to appropriate pockets too :)

Dear friends, as my friend had taken the world steemit be happy

Necesito seguidores @transisto ... se publican artículos bueno pero sino tienes seguidores tienes poco alcance. sera cuestion de tiempo? o necesito hacer post más explosivos? estoy atenta a tus comentarios

Woo hoo vote number 1500. I just hope my small vote helps enough for ya and good luck.


RESTEEMED & You got my VOTES!!! Keep up the great work. #MakeSteemitGreatAgain

You are setting a great example for us. We appreciate your time and investment into Steem. I hope more skilled people will jump in and help. In a few years you will look back and say it was well worth the effort. Thank you. Joy

I'd like to contribute!! Happy new year!!

hoe I wish my upvote can reward this post, but low Steem power, that won't stop my upvote still. @princekelly

I see another user who copied pasted the same post :|

I should to test various steemd configurations, to achieve desired performance and provision the appropriate hardware for the next 6 months of operations.

remarkable. Good luck

Glad new year! 2018 will be an incredible year. What an adventure it has been with Steem. We both have seen a considerable measure over the previous year or something like that. This undertaking has shown me to such an extent. The things I have learned are worth more than any measure of cash en route. I see the world in an entirely different manner after all I have learned since beginning with Blockchains. I truly trust the world is moving into a superior place. In the expressions of the colossal Naveen Jain we as a whole should "have an outlook of plenitude". This blockchain can be a device to change the world for good and show individuals numerous things if utilized as a part of the correct way. Information is power and It will be a blockchain loaded with learning. The Steem blockchain is getting more grounded by the day. We have seen some astounding development since this blockchain began and I know this is just the begin. When we have a completely refined item it will have the capacity to do things we cannot envision yet. The expansion of SMTs into the blockchain will intrigue and will change the way we consider the web and what is conceivable. Groups and discussions are a gigantic piece of the web. The conceivable utilizations of SMTs are stunning to consider. With Steemit and Steem we are now rolling out improvements in places that need it and offering assistance to those in require. Envision what's on the horizon for us and this blockchain. We additionally have an incredible group behind this blockchain. That is one of the principle reasons I put my BTC into this task over a year prior in the wake of discovering it on the Bitcointalk gathering and faltering onto Steemit through google look through a couple of times (accidentally hunting down different things). I at that point started some examination and saw @dan talking a couple of times about Steem. At that point saw @ned talking on YouTube and on CheddarTV and knew instinctively then he was resolved to make something extraordinary. Its awesome to see designers as you doing extraordinary things for the eco framework also @exyle with things like Steemify and Blockbrothers. Cheerful New Year again @exyle - Lets take Steemit and Steem to the following level in 2018!

@Furion I am new. Like, plankton new. Can you briefly explain what the purpose of replying with “comment #1/ 1.5” etc is and how it supports you other than just up voting and replying? Other than that, I am thoroughly enjoying this community and want to dive in to support!

I don't know how to take part in donation yeah I upvoted this post at least. This is my contribution , the most I could do. #Stay blessed @trassisto

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Thank you for all your incredible work!

.

You explain a lot of jargon that will take me a full day to dissect and understand.
This is good reminder and hope the best to get steemdata keep on going.
Keep on making steem the best crypto out there.

Every day of your life from now on to be good

Contribution comment #?
Hello there :)
Always great to discover new articles and I really like yours!
You can follow me if you like photography and if you are interested in seeing beautiful places, portraits and landscapes: as a passionate photographer I try to build my place here by sharing my work. I hope you'll like it :)
Here is an example of my work :
6_Photography-landscapes-nevada-swiss-photographer.jpg
and you can see my personal presentation here
Have a great day!!

Dear friend
Like getting a good graphics
I announced you
you are a The world famous grapher

give me vote and follow i also give you vote and follow

Generous!

thank you sir
you are very good luck

For whatever it's worth, I gave my 100 % upvote. Upvotes on this comments are going to be donated too.

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Nice post!

What is the delay between sync of steem blockchain and the mongoDB server(s) ?

I do not use Steem Data, but I can imagine that it costs a lot 🙈 Great innotiative as always @transisto 😍 upvotes and resteemed 👌

Dear friends, as my friend had taken the world steemit be happy

Dear brother I have benefited from your unlimited which does not end in.
Really you are very useful man
As a Liberal, like the sky, huge mountains, like the sea of the infinite mind.
Want to improve your world steemit unlimited, and pray to the Lord you're so long.

Contribution comment #000
heheheheheh,......

thank you dear friend
I want your heelp in steemit

Good info here. Thank you for posting! Happy new year

will you really Donating 5000$ @transisto ???
or just fucking said..

Contribution comment #000
heheheheheh,......