Donating 5000$ and raising money with this post for covering some of SteemData costs + server upgrade. (by @Furion)

in steem •  6 months ago

I commit to personally give 5000$ worth of STEEM or BTC at his convenience and 100% of the reward from this post and below comments to @Furion to cover server costs for hosting SteemData. Will also request from him some kind of proof of payments to verify costs.

What is SteemData?

SteemData helps developers and researchers build better STEEM applications. We parse the STEEM blockchain for you, and provide the data as a fast and convenient MongoDB service.

https://steemdata.com/

Here is the last update he made about SteemData




SteemData is currently available in a limited scope.

The following features are not available at the moment:

  • AccountOperations (Account History /w virtual ops)
  • Additional Accounts Data

Issues

Hardware

A full steemd node /w high-throughput is required for SteemData to function properly. It needs all plugins and all operations enabled, so that it can construct account history and store all the virtual operations alongside the operations stored on the blockchain.

Further, the node has to be in the same datacenter as SD to handle the required throughput. This is because the usage of Steem blockchain is growing, and re-syncing all the affected state requires over 100 requests per second. The private network latency within a data center is typically below 1ms, while a public network latency is usually 10 fold or more. This would decrease SD's throughput significantly.

Unfortunately, the datacenter where SteemData currently resides only offers servers with up to 256GB of RAM, and a steemd node configured to SD's requirements needs more than that.

Also, the SteemData MongoDB server is running out of disk space :(

Rusty code

I have patched things up in a quick-and-dirty fashion for a couple of months now, and the SteemData codebase has gotten a bit messy. I see this as an opportunity to clean things up, and improve reliability/performance.

Further, this is an opportunity to add infrastructure support and documentation, such that anyone can spin up their own SteemData cluster.

What needs to be done?

I am currently speccing out the new infrastructure. A new cluster will be setup in a different datacenter, capable of provisioning servers with NVMe SSD's in soft & hard RAID configs, and up to 512 GB RAM.

I should to test various steemd configurations, to achieve desired performance and provision the appropriate hardware for the next 6 months of operations.

The new DB server will also benefit from faster SSD's and larger in-RAM cache (currently 30GB, will be 60GB or 120GB).

Creating a replica set would add resilience and decouple BC processing from the database, freeing additional CPU cycles for queries. Further, adding replicas that are geo-distributed would allow for low latency in-app integrations worldwide.

As mentioned before, this is an opportunity to upgrade the operational side as well, with automated provisioning and monitoring/remediation improvements.

TL;DR: Wishlist

  • Clean up the codebase
  • Infrastructure as code
  • Documentation
  • New Servers
  • Multi-Replica DB Cluster

Thank you

SteemData is a popular choice for indie developers and power users.

Steem is growing at a rapid pace, and its daily on-chain state throughput is making Bitcoin and Ethereum look pale in comparison. To scale SteemData I'm looking at the approx. recurring server costs of $3,000/mo. This would not have been possible without witness pay. Thank you for supporting my work, and thank you for using SteemData

-- @Furion





Thank You.

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  
·
·
·

@transisto and @furion : what about data sharding the blocks chronologically and load balance from there? Of course I have no insights in steemdata's exact loads, but aren't the latest blocks queried most? If so, a relatively small chunk of data consumes most CPU / RAM but only a fraction of SSD, so by sharding "new blocks" (MongoDB can shard by default) on a separate node splits the SSD vs CPU/RAM/bandwidth problems.

Just an idea! ;-)
@scipio

EDIT: small self-upvote for visibility, 100% upvoted "contribution comment #4"

·
·
·

Scaling MongoDB is currently not a problem. The database would need to grow by another 1000% before sharding becomes relevant.

·
·

I think this is a very good in Amazon forest Amit user community don't forget to Upvote

·
·
·
·

Good Work Follow x Follow please <3

·
·
·

hola aca saludos desde venezuela me sigues te sigo :$

·
·
·
·
·

Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me

·

Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me

·

Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me

·

SteemData looks interesting might check it out on some future project. Not been 100% sure of current APIs.

·

@transisto and @furion So many great developments on the steem blockchain, and this is a great one to get more smart and creative developers on the block. I just upvoted both the post, and this comment.

By the way @transisto , I made a reply in your previous post of the size of a normal post, and your feedback is very important to me.

Here is a link to your previous post and my comment.
https://steemit.com/steem/@transisto/my-version-of-steem-is-not-content-based-it-is-contribution-based#@gold84/re-transisto-my-version-of-steem-is-not-content-based-it-is-contribution-based-20171230t142131356z

Looking forward to hear from you!

Regards, @gold84

·
·

yeah man there is so much demand for new deve;opers and great upvpyte potential for them that we should make a special front end just to post steem development projects and it will have a list of supporters, peopel who delegate steempower or curate themselves to peopel who post godo proejcts, kinda like @utopian-io for dvelopers, it would be really nice, it can allow steem to have a WHOLE github like system with slack as well, we should REALLY make the STEEM BROWSER and have a whole Steem Giuthub type website where tseem posts temselves hold the code, and just replace github repositories with steemit pots and comments

·
·
·

@ackza I totally agree with you! Thanks for adding this great idea to the message. Hope to hear from you more oftern on my post ideas/chapters. You have a lot to share there.

I am not sure if you know I am completing a full 11 posts/chapters steemit guide for new visitors, new steemit users, minnows, for them to learn as much as possible and as easier they can about steemit and the steem blockchain, to make their way here. In my already first 2 chapters published I got lots of questions, comments, ideas, and I believe it is helping them.

Looking forward to your thoughts when you get a chance.

Regards, @gold84

·

with this guys supporting steemit, steemit can only get huge!! we believe it

·

What is this contribution thing ? Guys

Explain me please

·
·

He wrote that all post rewards goes to funding the project including rewards from comments . So if I want to support them with more than 1 full upvote, I can both upvote his post and one or more of his comments. That's why.

·
·
·

Next time I'll stack them all like this so no random comment can fuck up the order.

·
·
·
·

Even if you stack them like this, this does not guarantee that others will not comment and break the order of your comments. And I guess the last comments won't be visible in the same page so there are sone disadvantages...

·
·
·
·

#1

·
·
·
·
·

#2 ...

·
·

I'm a bit confused too I guess you just have to upvote it. Nice to see people investing and growing the community as this is what I'm here for :)

·

DQmVXKKuHaYTK3VMKvswmon4p3ZcSv1iP3BY7Vd5kuB9toh.png

·
·

·

Good innovation upvoted

·

contribution comment #1.5

·

If you like the book, it is a request for reading
https://steemit.com/book/@saifuk/dan-brown-origin

·
·

Using the same comments that has nothing to do with the post over and over again may give you some reports. I can see that you mostly write good comments, so I wont report you this time, but please dont spam

·

Thanks...!

·
·

Comentario de contribución desde Venezuela, Excelente (Y)

·

Okie lets do it...
My Contribution Comment :)

·

@transisto It's such a great pleasure to be part of this contribution. I strongly believe the turnup is gonna be massive.

@tomsnoob

·

Contribution, contribution... Let`s go working

·

my last post contains a free gift Happy New year everyone!

·

Contribution comment #21

·
·
·

Great job.

·

please up vote my posts as u have strong steemit power

·

I didn't understood what is the work but still if you agree to do something good for steemit community .. this is worth commenting 👍

·

My last post contains a gift, Happy New year everyone!

·

Bro, how are you able to get this huge upvote. I am new on steem can you please show me the way?

·
·

Just for upvotes you can use @booster bot

·
·
·

How do I know it worked. Is it @boosterbot
Or
@booster bot

·
·

give me vote and follow i also give you vote and follow

·
·

voted and followed

·
·
·

Dont follow people that ask for followers without contributing to anything. He wont be a loyal follower, and he is spamming. The comment is copy paste

·

Keep up the good works..power!

·

Dear friends, as my friend had taken the world steemit be happy

·
·

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Great initiative!
I'm planning (as already mentioned in my last post) to provide some benchmark results for different node configuration to help make better decisions regarding infrastructure needed to run various Steem related services (so it would be for both consensus and API nodes and later whole infrastructure supporting it)

Server(s) costs itself even over-scaled and powerful enough to handle all expected load are not that big compared to infrastructure features such as HA, (redundancy, load balancing, self healing) etc etc and of course human labor.

Currently I would suggest using smaller servers with specialized steemds (consensus node for handling subset of calls, then account history (ops can be whitelisted differently for different nodes and different use cases) etc etc) Don't throw requests than can be handled by consensus nodes at your full nodes.

Maybe this has been asked somewhere already...

Why does Furion actually need more help with server costs? He has been one of the top witnesses for a while now and as one in the top-20, he is earning 7000-8000 SP per month. At current prices (~$2.85 right now), that's $20,000+ per month vs. ~$3000/mo. renting the equipment. Break-even on the new projected costs would be a $0.50 STEEM price.

For a few thousand dollars, you can buy your own 512 GB RAM server. Cash out a few thousand STEEM from witness rewards just from this month and he can buy two.

·

Not too long ago, Steem was at 80c.

You're right, I should just power down some Steem to cover the costs. I do however feel bad whenever I do that.

SteemData is well funded given the recent price increase and my witness position. I would like to forward the funds raised in this campaign to other Steem projects.

·
·

There’s no need to feel bad for powering down, especially since the witness rewards are received as SP. It’s the only option.

And just to clarify - I get the costs of your project. I just didn’t understand why ~$1000 of post rewards were needed when witness rewards are 20+ times more than that...per month (at current prices). If you cashed some out tonight, you could potentially get about $5 per STEEM. Seems like a great deal to me!

Good luck with your new servers and upgrades! I have a feeling that most of us will be upgrading witness specs again soon with STEEM prices continuing upward.

·
·

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

·

That's why I think, that steem does not scale.
512GB Node ..that is crazy. No one, except rich or already SP rich witnesses could afford running this
STEEM goes further and further away from decentralization. heading into a 20 Witness "diversivied" centralized system

·
·

Currently my most powerful nodes has 64GB of RAM.
Yes, steemd is very demanding. No, it does not need that much RAM. Optimizations are on their way (easy to spot while following github). Eventually, what we need from steemd itself works fine with 16GB of RAM.
What we are talking here is the infrastructure to provide SteemData service.
Servers are cheap. Smart humans are expensive, but worth their costs.

·
·
·

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

·
·

Steemdata is a backend that powers down stream applications. It will have processing and throughput needs over and above that needed to maintain a blockchain node. I’ve used the api and written big queries... it needs an upgrade

·
·
·

like getting a good post

·
·

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

·
·
·

I know..I am good!
Chers!

·

Hasn't he already crowd-funded some equipment before/

·
·

I’m not sure about equipment, but there was quite a few thousand allocated via posting rewards for SteemQ, which then turned into Viewly. And last I saw (3 months ago), there was to be an ICO for Viewly. Haven’t seen or heard anything about it since then. Furion’s social activity on the blockchain has only been one comment in the past two months...and that comment was worth $190+ when I saw it earlier today.

·
·
·

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

·
·

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

·

A single person running SteemData handles System Administration, Database Administration, Network Administration, Development, Quality Assurance, Testing, Planning, and Customer Service.

As a systems administrator and senior developer I was making over $100k/year. At the prices you detailed above, that puts the income around $250k/year. If you don't feel like you are worth $250k/year, or if you think anyone making that much should always spend their own funds and never get help from outside sources, then thats your opinion.

Maybe you just don't realize the amount of time, work, and effort this type of business requires. The upkeep, the research, the optimizations, the idiot users that fuck things up, etc.

Furion isn't just hanging around https://steemit.chat all day. I think he deserves well over $250-300k+/year

·
·

Then pay him the money you think he deserves. The difference between this blockchain and a job as various tech administrators is that those jobs are typically paid by a company with revenue streams from their products and services, or from a third party who needs the particular administration services.

If furion needs or deserves $300,000 a year, then perhaps he should work on monetizing his services rather than relying on charity from the limited daily reward pool. From a business perspective, that is what would make sense when it comes to expansion and covering additional costs, not a bunch of “contribution comments” on a post.

And since you’re up-to-date on what furion is doing on the daily, maybe you can tell us what has been happening with Viewly and its ICO? Or maybe let furion know that he can always drop an update on us when he’s not working at his eight SteemData jobs.

·
·
·

He is getting what he deserves. You are claiming he shouldn't. Just salty, I think.

But hey, it is quite alright that we have users posting 10-20 times a day getting $300 on each while not giving back to the community so let's just attack developers that are actually hosting servers and developing and maintaining them.

If you feel this post is overvalued, or any other post, you are welcome to use that downvote button and make that clear.

Now ... perhaps you have some kind of personal issue with furion. To that, I can say or add nothing to. In fact, I am unaware of anything about Viewly ... however, I will likely look into it a bit since you bring it up here.

·
·
·
·

Hi, our Telegram is always open for everyone to join. https://t.me/viewly . Thank you for your interest and kind words

·

wow excellent post

·

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

SteemData is currently available in a limited scope.

The following features are not available at the moment:

AccountOperations (Account History /w virtual ops)
Additional Accounts Data

Why not support SteemSQL in a similar way?

·

I don't think this is an either or type thing.

Both services are great and I personally have used both.

That being said, SteemSQL is a fucking beast and thank you for the service.

·

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

SteemData is really great and I am using it for some features for dMania.
Having more options to query the Steem blockchain makes a lot more features and apps on Steem possible. That is why a service like this is very important.

Lately it was lagging behind and the data wasn't up to date sometimes. I guess it was because of the limited hardware resources.
If SteemData is reliable and all the data is up to date, I think it will become crucial for the growth of Steem and we will see a lot of new great apps and features.

@Furion don't forget to add more indices for comments 😅 I really need them.

·

I agree with you we cultivate

·

Hi there @zombee I have been trying to post more on dmania so does that mean like I m doing something wrong ? I feel I can make people laugh. So I try to make more post ... Then today I realised not many people are looking into it .. n my post promotions went worthless ... So I m not sure should j make more post or just a few

·
·

dMania is still growing. Every post and contribution helps us grow. Thanks for your support.

·

& truth be told I m having zero knowledge of coding .. I can barely understand what to do with these kind of post ... But I try to searc keywords mentioned in these .. n m reading everything line 12 hours a day

·

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Contribution comment

This is some good stuff. I'm glad to hear that we are improving the infrastructure used by the tools used by the community.

Awesome. Voted!

interesting! I want to use SteeemData later

เห็นด้วยอย่างยิ่ง @nongboy
https://steemit.com/@nongboy

Once I want to get a fund for the 'Rohingya' refeuges in steemit ..... But I cant .. I am not big enough to do that.. Also I am not big enough on steem as well ... So I cancel the dream to raise the fund and help them what I can effort. But you doing a great job big brother. Help the people .. Maybe Allah help you. Insha Allah you get success in this work..

Data Maintenance and Server upgrade it is really costly. I appreciate about the work. Really it is talented work.

[Prediction] The next U.S. Presidential election will use blockchain technology and it will be totally transparent and fraud-proof!

Ok guys get start being new year dhamaka happy new year 2018 in advance

I always miss out on the contribution comments :(

·

You might not understand the point of them.

They are comments that will pay out to @transisto who will then donate the money to SteemData. They are not for random users like you or me to come to this post and get free money.

You should instead vote for the contribution comments to help contribute.

That is a very good cause for raising money!
Apparently, Steem rewards can go to appropriate pockets too :)

·

Dear friends, as my friend had taken the world steemit be happy

Necesito seguidores @transisto ... se publican artículos bueno pero sino tienes seguidores tienes poco alcance. sera cuestion de tiempo? o necesito hacer post más explosivos? estoy atenta a tus comentarios

Woo hoo vote number 1500. I just hope my small vote helps enough for ya and good luck.


RESTEEMED & You got my VOTES!!! Keep up the great work. #MakeSteemitGreatAgain

You are setting a great example for us. We appreciate your time and investment into Steem. I hope more skilled people will jump in and help. In a few years you will look back and say it was well worth the effort. Thank you. Joy

I'd like to contribute!! Happy new year!!

hoe I wish my upvote can reward this post, but low Steem power, that won't stop my upvote still. @princekelly

I see another user who copied pasted the same post :|

I should to test various steemd configurations, to achieve desired performance and provision the appropriate hardware for the next 6 months of operations.

remarkable. Good luck

Glad new year! 2018 will be an incredible year. What an adventure it has been with Steem. We both have seen a considerable measure over the previous year or something like that. This undertaking has shown me to such an extent. The things I have learned are worth more than any measure of cash en route. I see the world in an entirely different manner after all I have learned since beginning with Blockchains. I truly trust the world is moving into a superior place. In the expressions of the colossal Naveen Jain we as a whole should "have an outlook of plenitude". This blockchain can be a device to change the world for good and show individuals numerous things if utilized as a part of the correct way. Information is power and It will be a blockchain loaded with learning. The Steem blockchain is getting more grounded by the day. We have seen some astounding development since this blockchain began and I know this is just the begin. When we have a completely refined item it will have the capacity to do things we cannot envision yet. The expansion of SMTs into the blockchain will intrigue and will change the way we consider the web and what is conceivable. Groups and discussions are a gigantic piece of the web. The conceivable utilizations of SMTs are stunning to consider. With Steemit and Steem we are now rolling out improvements in places that need it and offering assistance to those in require. Envision what's on the horizon for us and this blockchain. We additionally have an incredible group behind this blockchain. That is one of the principle reasons I put my BTC into this task over a year prior in the wake of discovering it on the Bitcointalk gathering and faltering onto Steemit through google look through a couple of times (accidentally hunting down different things). I at that point started some examination and saw @dan talking a couple of times about Steem. At that point saw @ned talking on YouTube and on CheddarTV and knew instinctively then he was resolved to make something extraordinary. Its awesome to see designers as you doing extraordinary things for the eco framework also @exyle with things like Steemify and Blockbrothers. Cheerful New Year again @exyle - Lets take Steemit and Steem to the following level in 2018!

@Furion I am new. Like, plankton new. Can you briefly explain what the purpose of replying with “comment #1/ 1.5” etc is and how it supports you other than just up voting and replying? Other than that, I am thoroughly enjoying this community and want to dive in to support!

I don't know how to take part in donation yeah I upvoted this post at least. This is my contribution , the most I could do. #Stay blessed @trassisto

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Thank you for all your incredible work!

.

You explain a lot of jargon that will take me a full day to dissect and understand.
This is good reminder and hope the best to get steemdata keep on going.
Keep on making steem the best crypto out there.

·

The main thing to understand is that Steem nodes are expensive and furion could use some help for his great service

·
·

Dear Friend
Like getting a good post
I am very poor in steemit life
please follow & give me upvoted
your vote for me very needy in the steemit life

·

Every day of your life from now on to be good

Contribution comment #?
Hello there :)
Always great to discover new articles and I really like yours!
You can follow me if you like photography and if you are interested in seeing beautiful places, portraits and landscapes: as a passionate photographer I try to build my place here by sharing my work. I hope you'll like it :)
Here is an example of my work :
6_Photography-landscapes-nevada-swiss-photographer.jpg
and you can see my personal presentation here
Have a great day!!

·

Dear friend
Like getting a good graphics
I announced you
you are a The world famous grapher

·

give me vote and follow i also give you vote and follow

Generous!

·

thank you sir
you are very good luck

For whatever it's worth, I gave my 100 % upvote. Upvotes on this comments are going to be donated too.

·

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Nice post!

What is the delay between sync of steem blockchain and the mongoDB server(s) ?

I do not use Steem Data, but I can imagine that it costs a lot 🙈 Great innotiative as always @transisto 😍 upvotes and resteemed 👌

·

Dear friends, as my friend had taken the world steemit be happy

·
·

Dear brother I have benefited from your unlimited which does not end in.
Really you are very useful man
As a Liberal, like the sky, huge mountains, like the sea of the infinite mind.
Want to improve your world steemit unlimited, and pray to the Lord you're so long.

Good initiative, upvoted and resteemed.

·

Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH

Contribution comment #000
heheheheheh,......

·

thank you dear friend
I want your heelp in steemit

Good info here. Thank you for posting! Happy new year

will you really Donating 5000$ @transisto ???
or just fucking said..

Contribution comment #000
heheheheheh,......

Nice post 👍 Thanks for sharing and and very useful information and good job. @transisto

·

Dear friend
Arabica coffee to me very dear

Contribution comment #1337