BlockTrades RocksDB proposal to Steemit

in blocktrades •  7 months ago 

Steem nodes are costly to operate and the cost is increasing

The Steem blockchain currently requires significant resources to run a Steem node for acting as a Steem witness or for operating a cryptocurrency exchange. For example, if you want to run a witness that runs on an average cloud server, you can expect to need 47GB currently, and this is likely to increase beyond 64GB before too long. This latter value is important because server costs increase dramatically as a server exceeds certain memory requirements (64GB, 128GB, 256GB, 512GB). These memory costs also impose limits on the ability for the Steem network to grow in terms of users, posts, comments, etc, so it’s important to Steem’s future to reduce the costs of running a Steem node.

Replacing chainbase storage with rocksdb storage to slash node operating costs

BlockTrades believes the most suitable solution to this problem is moving much of the blockchain state data that is currently stored in a database called “chainbase” into a more efficiently organized form using the RocksDB database. Chainbase was an early attempt to reduce the memory requirements of running a Steem node, but it ultimately failed because the storage mechanism was poorly tuned for storing data on a hard disk, which lead to most nodes storing their chainbase in a form of “RAM disk” in order to operate at acceptable performance levels. In other words, node operators found that the data still needed to be stored in memory in order for the node to operate acceptably.

Unlike chainbase, RocksDB is specifically designed for efficient storage and retrieval of data using standard hard disk technology, so it will enable this state data to be moved out of memory once and for all, resulting in reduced costs to run Steem nodes.

Why BlockTrades is the natural candidate to implement rocksdb in Steem

The BlockTrades development team has long been concerned about the increasing costs of running Steem nodes and we approached Steemit over a year ago with a design architecture for using Rocksdb to solve the problem, based on our prior experience with high performance database coding. Steemit was generally receptive to the idea and we implemented an initial proof-of-concept under contract with Steemit that stored database from the “account history plugin” that carried one of the highest memory costs for the API nodes operated by Steemit. This technology was successfully deployed in Steemit’s production servers and has allowed Steemit API servers to continue operating on high end cloud servers.

Note, however, that this was just a proof-of-concept to show that Rocksdb met the performance requirments to allow state data to be moved from memory to disk storage by writing special case code to move the account history data. It did not reduce the memory requirements of the “consensus” state data that requires witnesses to run nodes with a large amount of memory. But we proposed a second stage project whereby we would develop a generic way to move Steem state data (stored in C++ data structures called multi-indexes) on to disk storage using Rocksbd, then use this technology to move most if not all of the state data onto disk.

An important part of this latter proposal was to also develop an extensive test framework to verify that functional operation and performance of the Steem node was not degraded as we progressively move more and more of the state data onto disk storage. At the time, Steemit rejected that proposal as they decided that enough memory optimization had been done for their immediate needs and Steemit planned to move all the non-consensus data out of the nodes into a separate database (variously SBDS and Hivemind).

If hivemind is going to reduce memory costs, why is rocksdb needed?

Hivemind will move “non-consensus” state data out of the API nodes operated by Steemit that are used to serve data to steemit.com and other web sites and user interfaces like busy.org, steempeak, partico, appics, etc. But it won’t reduce the amount of memory required to operate the most important nodes required to keep Steem a decentralized network: witness nodes. Rocksdb, however, will allow this data to be moved to disk, allowing witness to operate nodes at reasonable cost now and in the future as the blockchain’s activity level increases.

Why re-propose this now?

Steemit has recently expressed a renewed interest in reducing the costs of operating Steem nodes. At the same time, they’ve reduced the number of employees able to complete various Steem-related projects. We at BlockTrades believe that we can provide the expertise to complete the Rocksdb at an efficient cost and free up Steemit’s blockchain programmers to focus on other high priority tasks like implementation of SMTs.

What are we proposing?

Our proposal is to analyze the key data structures currently consuming memory in consensus nodes (i.e. witness nodes) and then move some of this data into Rocksdb storage using a generic solution we will develop. We will then thoroughly test the resulting version of the node versus the operation of existing chainbase nodes to ensure that the replacement will be seamless. The generic solution we develop can then be used to move more state data out to disk as required in the future and the regression test system we develop will be re-usable as well. We believe we can perform this work in 2-3 months with a 6 person team for a fixed-cost of $250K.

If this is an offer to Steemit, isn’t this better done directly to Steemit?

We have approached Steemit with this offer and they are considering it, but Ned asked that I also share this information with the whole Steem community and see how the idea is received. I believe this could a great first step to expanding development of Steem’s blockchain code beyond Steem’s core team and will enable Steem to progress faster than ever before, so please let us know in the comments how you view this project proposal.

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

In the era of automated upvotes, my 100% vote for this idea might not be a clear message ;-) but I'm all for this.

Some time ago I was hoping to move to std::allocator, which combined with a deterministic snapshots to save/load state will help a lot in maintenance of steemd nodes. That was before RocksDB, which is now not only a well proven solution, but also opens our way for some more fancy features, and all that without re-inventing the wheel (fast persistent storage).

It will help not only to reduce costs of API nodes (they will eventually move into steemd + bunch of microservices) but more importantly - consensus nodes.

We really don't want the exchanges to abandon us just because our state (file) is fat and ugly.

2-3 months, however, sounds very risky (I'll be guessing twice as much), especially that it needs to be tested throughly.

·

You would all be very wise to listen to this inhabitant of your planet.

If you need some assistance finding new ways to motivate these people to embrace this proposal, I am willing to lend a hand.

Specifically this one...


·

I think if they work together with Steemit's team, they can speed things up.

·
·

Do you know why Steemit isn't accepting outside help?

"Ned asked that I also share this information with the whole Steem community and see how the idea is received." Everyone seems to think this is positive offer and yet he decided to decline it... Now he's been silent for 16 days after not doing a promised livestream and has been in full power down for quite some time. What's happening?

·
·
·

Nobody knows what's going on in his head 🤷‍♂️

·
·
·
·

Shit, sounds bad. If only we'd had Vitalik with us

·
  ·  7 months ago (edited)

Evergreen content reduces spam thus reduces ram. The exchanges would abandon Steem because of low trading volume, Ned paid a pretty penny to get listed on binance, did you know that? He paid money. Over 100k, and that pays binance's nodes. Also ned delegated them. It seems the only people trading are the whales trying to dump on each other. Why would I ever invest into a coin the owner of majority (ned) is powering down? Many investors think the way I do.

Sounds good to me. Lets go for it.

I can't for the life of me see a downside to this... It makes absolut perfect sense to me. Not only does @blocktrades already have a dev team that is very familiar with the STEEM ecosystem, but more importantly this frees up Steemit Inc to work on SMTs.

And let's face it, many projects on here were basically betting on that horse to move their initiatives forward. This is the case for appics, actifit, 1up and the list gets quite long.

I sincerely hope this is agreed upon, and that Steemit Inc commits to keeping their initially announced timeline or get as close as possible.

There has been many comments regarding the importance of STMs to restore confidence, which let's face it, without, it's seriously hard to build anything of substance here.

So to me this is not a yes... this is a HELL YES! - let's make this happen, let's get these costs under control, lets move away from depending of Steemit Inc to do absolutely everything.

When the costs become reasonable as to allow us to run a node ourselves (in helpie or with a partnership) - I fully intend to throw our hat in the ring. To me it's never being about the idea of "Oh, but th node doesn't make me money" - I don't care about that, never have... However, I don't want to become homeless while attempting to afford one.

·

I will believe SMT exists once code is open source and we are testing on the testnets. If anyone banked their life or company on words uttered by steemit inc they made a critical flaw. Ned said back in 2017 SMT would be here in jan 2018. Wait. it's not January?

·

Fact is: 250$k is a lot of money. But another fact is: time is money.

2-3 months delay on SMTs is what most stakeholders of Steem are worried about.

By outsourcing the work to blocktrades, Steemit Inc. could get back on track with SMTs and depending on the need, could even release basic SMTs ahead of the scheduled time of May 2019.

Now, with having my limited knowledge in mind of how much Steemit Inc. can afford at this point in time, I'm still very much in favour of this proposal. However, in the end, Steemit Inc. has to make the decision if they believe it's a good investment.

·
·

Fact is: 250$k is a lot of money. But another fact is: time is money.

the simplest yet most realistic way of looking at this... Priority #1 is to restore confidence and that can only be done by trying to meet the already established deadlines.

·
·

I agree with everything except the last part of your comment @therealwolf since I think @ned and @steemit will factor in the support or non-support of the Steem community for this agreement.

·

this frees up Steemit Inc to work on SMTs.

Haven't they learned that SMTs should NOT be the priority for the blockchain?

·
·

Very valid point my friend, however the carrot has been dangled long enough and there is something that I can only describe as "hope fatigue" - Investors, both those in here already and those from the outside looking in have lost a lot of confidence. Delivering on promises is all about restoring that, even if it brings an avalanche of shitcoins, which it very well might.

·
·
·
  ·  7 months ago (edited)

SMTs will add more pressure and complexity to the system. If the system isn't scalable and optimal, we'll end up with more problems. They should have focused on that before making bold promises. Simply put, we're not ready for SMTs.

·
·
·
·

Part of the issue is that community features are tied up with SMTs, meaning that both the economic AND social feature set of Steem are held back by now launching SMTs. It would be great if somehow we could at least have communities get a chance to take root, even if SMTs take a lot longer. Communities should be a significantly simpler objective to realise.

·

I think it is a huge assumption that SteemIt, Inc. Will go back to working on the SMTs. Just keep in mind that is coming from your sense of priorities and not Ned's.

You have my vote!

I love the idea and the proposal. I also know what blocktrades is capable of and have full confidence in their ability to perform.

Question: could part of the cost be offset by the community using its stake to upvote brief daily progress reports?

·

Realistically, I'd probably need to write those reports personally, and giving my own billing rate (much higher than our other programming staff), daily posts would probably end in a net loss, unfortunately. It's also difficult to describe the daily work done by a programming team in a form that is meaningful to non-programmers. I recall this very well, as I had a similar request made when we were working on the BitShares p2p networking layer many years ago (same one used now by Steem code base).

·
·

True.

In any case I have seen comments from several in the community that would be willing to contribute to this much needed solution. With it we all win. Without it............

If necessary I'd prefer to use my stake in order to contribute using whatever method would be the most practical.

I am all for this.

This is really what is needed. Other people outside Steemit to step up and do what needs to be done. I like the idea of your team handling the database conversion to best handle the nodes while lets the Steemit team to continue to focus upon SMTs.

That is a win all around if you truly believe you can pull it off, which I presume you do or you never would have proposed it.

Great idea.

·

Agree!

I have more confidence in your team’s development abilities than Ned’s. I would support this proposal.

How are you planning to cover the costs of development? Will this be paid by Blocktrades company funds or are you looking for payment from Steemit, Inc. or other blockchain-related options?

[EDIT] - Just read some of the other comments and found an answer. Will you still go ahead with development if Steemit, Inc. does not accept your proposal? You mentioned that this is a benefit for Steem and node operators, so could development still be possible if they reject the offer?

·

I'm willing to fund the effort in the short term as proposed, if Steemit will agree to some kind of reasonable performance contract, but I don't want to end up in a situation where our guys create code, their guys create competing code, then they just decide to go with their code cause "it's ours, it must be better" (and in any event, it results in a lot of lost programming effort).

If we deliver a product that provides real value to Steem and Steemit, I think it's only fair for Steemit to make a reasonable payment, as they benefit far more than we do. We're actually quoting at below our "normal" rates for fixed-cost blockchain work, as we do have some stake in the results and we hope this can lead to more opportunities in the future.

Some people have also suggested crowdfunding all or a portion of the work costs, but I think the expense and time overhead make that option less attractive in this particular case. That said, I do think it would make sense in the long run for Steem to have some kind of worker proposal system like BitShares has. It's been very successful, IMO.

·
·

Nice to finally see some action with my own eyes.. Yes yes yes make this happen!

Also, I don't know how not to get behind the idea of a worker proposal system like Bitshares. It has worked for them, it has worked for Dash, it will work for Steem, Power to the community.

Posted using Partiko Android

·
·

"I do think it would make sense in the long run for Steem to have some kind of worker proposal system like BitShares has. It's been very successful, IMO."

This!

good solution by ned scott

I fully suport the idea.
Steemit Inc can't do everything.
I would love to see the experts try to estimate time and effort to analyze if the 250k is a fair price for that work.
From my point of view if it is paid after the work is done it is worth to accept it.

Nice proposal @blocktrades
Really appreciated.

I have read all the comment and made me a picture.

BlockTrades has my voice to do it

This is a win win for Steemians and it creates room for investors

Any initiative that makes witnesses more affordable (ie not requiring potential memory pools of 128GB+ with scalability) is an excellent proposal. This will allow more competition in the witness game, and further increase decentralisation.

It would motivate someone like myself to invest in physical hardware, and then co-locate it in a data centre, instead of the current trend of people hosting servers on leased equipment, which is ultimately not "their property".

It feels as though having all of these "decentralised" witness servers (many of which are physically located or hosted by the same providers) - presents a risk to the blockchain - what if that server provider was shut down?

I'd love to see more witnesses / full nodes in locations that are not data centres for these very reasons:

  1. Increased decentralisation
  2. Increased competition = witnesses that do stuff for STEEM
  3. The concept of custody and ownership makes people care about things a lot more
·

that is why we at @swisswitness have our own machine located in a data center safely in the Swiss mountains... no cloud, no leasing
its a little bit more work to set it all up but was worth it in our opinion for all the reasons stated above. It helps with the whole decentralisation and as such makes steem more secure as a blockchain.
I know you have your 30 votes spent but if there is ever one that is freed up we would appreciate the vote.

·
·

Gave you the vote.

·
·
·

I have not seen it on the list maybe something went wrong. Here is the easy steemconnect link.
thanks for wanting to though :-)
https://steemconnect.com/sign/account-witness-proxy?proxy=swisswitness&approve=1

·
·
·
·

Check my steemworld.org and you'll see I did it

Posted using Partiko Android

·
·
·
·
·

yea, its weird, i usually use https://steemian.info/witnesses to check and it was not there but I can see it on steemworld and on steemd.
thanks for the vote, its really appreciated

This would be fucking incredible! Here's hoping it turns into a thing!

This solution sounds to be significantly better than purely relying on the previously presented concept of modularising the node software to facilitate clusters of servers that operate as a single server (allowing for increased levels of raw processing power - but with greater overall hardware cost).

I imagine the quoted cost would be more than covered by less than a year's operating with the more efficient system and the associated lower hardware running costs. I have no idea of the state of Steemit Inc's 'bank account' - but one way or another the community can afford this.#

It should also be highlighted that @blocktrades has stated that if Steemit inc. doesn't accept the finished result then they don't pay.

·

Good point about reducing cost versus spreading the load over multiple servers via the modular blockchain idea. The modular idea has it's place, but the low hanging fruit should be gathered first.

·
·

Absolutely, yes!

·
·

I approve this message, and proposal.

Thanks for outlining the proposal. A store developed, hardened and proven is better than rolling their own.

  1. Do you expect that there will be significant in-RAM data that can be moved into the storage?
  2. Do you think there might be space savings to be had by reducing duplication of objects in the store? It appears that between the consensus data and the data needed by the APIs there is the potential for a lot of unnecessary duplication.
  3. RAM storage is typically treated as synchronous to access, RocksDB would be async. Do you think that will add risky complexity to your proposed solution?
    Finally,
  4. Would @blocktrades be open to a performance-based contract? ie most of the payment made for acheiving a specified maximum RAM usage at a certain lookup performance on a certain CPU / RAM / storage config?
·
  1. Yes, pretty much all blockchain state data is currently stored in-RAM (with the exception of account history for nodes already running with the rocksdb version of that plugin). So long term we can expect the RAM requirements to drop to a very small amount, since most of the data is stored in those multi-indexes.
  2. Good question, we actually did look into that a while back and saw some oportunities for moderate gains from reducing duplication. But those gains only really become attractive after we first attack the "elephant in the room" that currently dominates in terms of RAM costs (chainbase itself).
  3. Yes, and our architecture includes a caching design to address such issues. But obviously we have to still deal with cache misses, and this is why we've placed such an emphasis on developing a rigorous testing system to catch problems that could result.
  4. We would, but first we need to get the multi-index usage data from Steemit (it exists, but our team no longer has access to it) so that we can figure out a reasonable plan for what things we could most safely move out of memory and for what benefit. If Steemit give the nod to this idea, we can get that data, make a quick estimate on the memory savings, then write the contract terms based on that.
·
·

Thanks. You have my full support.

·
·
  1. Would @blocktrades be open to a performance-based contract? ie most of the payment made for achieving a specified maximum RAM usage at a certain lookup performance on a certain CPU / RAM / storage config?

I think a better payment model would be paying a much smaller amount up front and the balance over time out of the savings achieved. It is in everyone's best interest to not over spend Steem Inc's remaining resources while the token is so low.

I’d also note that these RAM levels correspond to motherboard & chipset limits in the real world.

  • 64Gb is the RAM limit for a regular desktop computer (eg LGA1151 socket);
  • 128Gb is limit for High End Desktop (HEDT) like Intel Skylake X CPUs.

Beyond 128Gb RAM you have to use server motherboards and components which are often much more expensive for small performance improvements.

For true decentralization we need to be in the situation where witnesses can run on regular desktops with 64Gb and full RPC nodes on HEDTs with 128Gb.

Preferably on people’s own hardware as use of cloud services undermines decentralization and creates ongoing operational expenses that can be problematic in times of low Steem prices.

Purchased equipment means you can ride out the tough times. It also proves your commitment.

Posted using Partiko iOS

·

Youre right, the main problem is not beeing able to run a node on a "more regular" machine, as this contributes to more centralization!

·
·

Hell, if I could run a node out of my basement on a decent desktop, I would. Just saying.

·
·
·

Same here, or even on a cheap aws server!

does anyone need some heavy duty onsite equipment? I have connects to one of the largest IT resellers.... Dell HP Lenovo... anything you want... I probably can't sell it to you but I can help you get pricing and connect you with the rep who has your territory

While I am extremely supportive of this in concept, I would like to see more details about the work being performed and how the time and cost estimate was calculated. A 6-person team for 2-3 months is a pretty big project and $250k is a significant expense.

I would ask to see a detailed task breakdown of the work to be performed and what exactly each of the 6 people will be doing for those 2-3 months.

·

We could do it for less if it was being done in a way that was risk-free to us (paid by time/labor required rather than a fixed price contract). That's actually the way we do most contracts and I rarely offer a fixed-price contract because of the difficulties of accurately predicting the real costs of software development. I'm giving Steemit a fixed price option in this case so that so that they have a clear upper limit on how much the cost for the work will be. Based on the ROI, I think it's a good offer.

A full breakdown of the actual work isn't really easily recreatable by us until we get more into the task, nor do I think it would be of much interest to most people, as it would very quickly get extremely technical. We have shared such data in the past with Steemit (we don't have access to those documents now, they're owned by Steemit).

·
·

I'm looking at this from the perspective of if I were steemit, inc and I was evaluating this proposal (which I hope that they are). I would just like to see more details is all. You must have done some type of task listing and estimating to come up with the 6 person team size and time frame so I think that would be good to see.

Overall I'm excited at the prospect of having your team be involved in development. It was great to meet you at SteemFest and given your large stake in the platform I'm confident that you have the best interests of Steem in mind, which is something that we cannot say about most external developers that we might look at working with.

·
·
·

@yabamatt, for those of us Steemians who don't know who the "I" is speaking for BlockTrades and the corresponding "you" in your response, would you please enlighten us?

Obviously very serious matters, if not the long-term survival of the Steem blockchain, are apparently being discussed, in which case I would hope we can get all "out in the light."

"All cards face up on the table" is another way we might say the same thing here in America.

I appreciate you pressing for details, given all that is at stake (pardon the pun) ... 👍

Posted using Partiko Android

·
·

The fixed price until solution delivery is a ballsy proposal, usually during such projects unforeseen challenges arise. I hope you do not expect to make a lot of profit on the project itself. It could be a great value-driving improvement to the Steem eco-system though.

Is Steemit paying or should we crowd-fund with the community? Maybe 75/25%?

·
·
·

Yes, it's a bit risky, but as long as we can get an acceptable scoping agreement with Steemit, I think its a win-win.

Currently the proposal is for Steemit to pay for it. Crowdfunding is another option, but I don't know how much overhead is involved in going down that path and I hope we can jump start this project quickly.

·

If you look at it purely for paying the staff involved, that comes out to $13,000 per person per month. Not including software licensing, overhead costs (electricity, internet, office space, etc), and that stuff.

Actually, in looking at the responses, a crowd funded option that @fitzgibbon suggests isn't a bad idea. People invested in Steem's success would likely be willing to pitch in...

You have my vote. Your team seems to get things done efficiently.

And this is exactly why I love having voted you guys as my witness. Taking initiative. See a problem, and propose to fix it.

I've don't know much about running a witness server, or anything in general and would like some things clarified, and hopefully that helps others out too.

  1. Is the 47 GB thats needed SSD storage or RAM?
  2. You stated that the work could be completed for about $250k. Do you have an estimated amount saved for STEEMIT INC.?
  3. Would STEEMIT INC. be paying that $250k? After laying off 70% of their staff, do you think they would be willing to do that?

Again, thanks for being awesome and supporting STEEM out in multiple ways. This is what I hope to see from more witnesses.

·

Great questions...

  1. In practice, it's RAM. We tend to run our Steem nodes on 64GB RAM virtual machines.
  2. I'd prefer not to make estimates for them, but I'm confident they would rapidly get their ROI (within less than a year).
  3. Yes, Steemit Inc would be paying the $250K (at least under my proposal). And they can easily afford it, IMO. The layoffs were done to reduce their enormous burn rate, not because they were facing anything like imminent bankruptcy. Steemit is in a strong financial position right now, IMO.
·
·

Thanks for the response. And the savings would be shared by the witnesses too right(due to lower server requirements)?

·
·
·

Yes, it would allow witnesses to run cheaper nodes now and even more importantly, in the future, as Steem blockchain activity levels increase.

·
·

Great price IMO!

·
  ·  7 months ago (edited)

Is the 47 GB thats needed SSD storage or RAM?

Currently the shared data area for a witness node is about 47 GB however the data area doesn't all need to be in RAM. It seems 32 GB is enough RAM to run a witness node with state data swapped from SSD by the OS and even less is possible (I've successfully tested as low as 4 GB, though that was a while ago). Low memory configurations carry a severe performance penalty and operational challenges and may become entirely unworkable should the network traffic increase.

A well designed and production quality database (which chainbase is not) is generally designed to get better performance with less RAM.

·
·

Agree with all of the above, naturally. But the performance penalty for operating in such a mode is too high IMO and will get progressively worse with time, based on current trends.

·
·
·
  ·  7 months ago (edited)

32 GB is not that bad still. My seed works with 32 GB with only a modest penalty. 4 GB is of course extreme penalty. I don't suggest anyone do that for production, and I only did it to see if it were possible. It was possible but not practical.

And of course RAM usage will continue to grow, although the budgets allowed by RC result in a fairly modest growth rate. That's a huge improvement where we were pre-HF20, when there was a risk of a rapid RAM blowup. There isn't now.

·
·
·
·

Yes, and as you probably remember, this was my primary reason for being in favor of RC (and also being opposed to previous suggestions prior to the RC hardfork where the witnesses could have increased blocksize to give everyone more bandwidth).

But while we're not currently risking a rapid RAM blowup, the new rules do put limits on how fast the platform can grow, and I believe that a full rocksdb implementation could alleviate this problem to a large degree.

·
·
·
·
·

Agree on that.

·
·
·
·
·

The reward pool is only so large, where is the tipping point on users vs rewards?

With stinc's delegations out the pool is already hit, most newbs are wasting their time.

This wishful thinking of taking fedbook's 100m users is unrealistic.
The math doesn't support it.
We should stop that pie before it gets into the sky, imo.

·
·
·
·
·
·
  ·  7 months ago (edited)

Steemit was never much like Facebook (the nature of it is entirely different), but other social sites like twitter, reddit, or even youtube are better fits as the sorts of models that Steem/it could seek to emulate.

That aside, I'm not sure what you mean about the reward pool only being so large. That, to me, seems to assume that the price of STEEM is a constant. With more users the theory goes that STEEM would be worth more. Maybe it would, maybe it wouldn't. I don't think you can view the reward pool as a fixed amount that inevitably gets divided up into smaller and smaller pieces with more users though.

I don't personally see anything wrong with engineering the blockchain so that it can support more users and usage and to do so with lower costs. Indeed it seems to be a very good thing.

Loading...

Sounds like a rockin' good plan to make things run smoother for everybody. Thumbs up.

One Word Answer NO.

We don't need to spend more of Steemit Inc's Money, which is essentially all of the stakeholders equity as it takes steem to pay blocktrades to do this. This is sending good money after bad.

What we need is Steemit Inc to focus on blockchain development and sunset steemit.com and all of the nodes and let the apps themselves find a way to run their own nodes in a survival of the fittest environment.

What is proposed here is more centralization which is what scares off a lot of people and investors to buy steem in the first place.

Big no thanks from me.

Btw it's pretty obvious Ned has a problem saying NO, that's why we are in the situation we are in now.

I should add that steemit.com could always be brought back at a future point in time when things are more stable.

Self sufficiency is what we need not cashing out more steem to pay blocktrades which further decreases the price.

·

These changes are necessary to increase decentralization. The RAM requirements for both witness and full nodes are too high and thus make running them too costly for most people.
Getting costs down is crucial for true decentralization as I’ve detailed here: https://steemit.com/steem/@apshamilton/decentralisation-of-full-nodes-essential-for-steem-s-success
If Blocktrades can do this and it lets Steemit Inc focus on core blockchain development I’m all for it.

Posted using Partiko iOS

·
·

It's a big IF, and costing up to 2.5 million steem isn't a drop in the bucket.

Also how do we know for sure this will work and not create more issues like slowing down the applications?

·

If I understand your argument correctly (and maybe I don't), you seem to be assuming that Steemit won't spend the Steem on anything else if it doesn't fund our proposal. But if so, I have to disagree with this assumption: I have no doubt that over time Steemit will spend its Steem and given past history, the amount we're asking for this contract is a drop in the bucket.

The only question is what it spends it on. It can spend it on internal development of the Steem Blockchain, external blockchain development (i.e. contractors like us), new products of an unknown nature, or Steemit investor payouts. Steemit's internal development team is pretty much maxed out in terms of what it can do now and the only way they could do more would be to re-hire more programmers again, and that's costly and risky (it's difficult to know what you're getting with new hires and it takes a long time to train them to the point they become productive). Also, hiring more internal developers seems much more centralized to me than building up experienced external teams which can take on 3rd party projects in the future.

Maybe it needs emphasis that this work isn't aimed at making steemit.com able to operate more cheaply, although that would certainly be a side-effect. The major goal is to reduce the cost for 3rd party node operators (witnesses, exchanges, and even full API nodes). This is also means that competing 3rd party front-ends won't need to rely on Steemit's API servers to operate (i.e. the network becomes more decentralized). It's a one-time cost that brings down the costs to allow new Steem businesses to compete without leaning on Steemit's infrastructure.

·
·

The 250K is a drop in the bucket thinking is what got Steemit Inc into the position that it's in now. No thanks. Also you're assuming that you're the only one that can do this or that your code is somehow the best. Let businesses create their own nodes, they can prob find a better solution than yours anyway. We need a lean Steemit Inc that respects its cash reserves.

·
·
·

No, it's not the kind of thinking that got Steemit into the position it's in. In point of fact, if Steemit had taken us up on our earlier offer, the work would have already more than paid for itself by now. The failure was in thinking that the ongoing costs weren't that significant (crypto prices were higher and node operational costs probably didn't look so important that they needed to be addressed in the near term). Also they wanted to keep most of their development in-house, despite not being able to hire enough qualified blockchain programmers internally to address the solution themselves.

We're not the only one that can do this. I do believe we can do this more efficiently than any other party out there, in part because we have one of the largest teams of experienced blockchain programmers out there, and we already have a lot of experience with Steem and rocksdb in particular. Also, we already invested a lot of time in working on this solution as part of the proof-of-concept work, which gives us a further advantage. I don't know of any other company that can say the same.

If you think there's some business that can do this right now, other than Steemit or us, I suggest you contact them and see what they say.

·
·
·
·

If steem goes to 10c which is easily possible in this bear market, that 250K would be 2.5 million steem. That is not a drop in the bucket.

Maybe it would have been better maybe not it's in the past. I still think steemit.com is a huge waste of resources, and it would be better to sunset it and make it a wallet, while letting the businesses themselves come up with solutions. Innovation of nodes will def happen a lot faster when steemit stops giving them away for free, because the businesses will be forced to make it profitable themselves. And whichever one succeeds will end up getting the most traffic which means they make the most money.

Just because there isn't a team out there right now that can do it doesn't mean there will be once attention is put to the area.

Btw, experience doesn't matter that much to me. If you look at google, they hire much more based on ability than experience.

·
·
·
·
·

this all seems very pessimistic to me. I would say that the proposal is a good one.
There are a lot of "ifs" in your assumption.

Steem holds about 60 million Steem that is at current prices 20 million dollars. So what "if" that 2.5 million goes to blocktrades? they support the community and are here to make the blockchain better. Its not like they are going to dump it on the market??
If by spending 250k they could reduce the cost of running a witness (there are about 200 witnesses out there not counting the backups) that would be saving a lot of people a lot of money. AND they would save Steemit Inc a lot of running expenses that they claim would be roughly won back in a year. Then why the negativity?
And why would steem need to keep their 60 million Steem?? If they can cut costs and cut spending through this solution its a good one. Plus I would say that @blocktrades would be held accountable by Steemit inc and by the community as a whole if they do not deliver and try to take advantage of the whole situation.

·
·
·
·
·
·

There's a lot more "ifs" in hiring blocktrades than not. It's only a promise and no guarantees. Coding software doesn't always go as planned. I see it like this either

  1. Pay up to 2.5 million steem for a hope

  2. dramatically reduce costs and let the businesses innovate for themselves and focus on what steemit inc does best

I go for option 2. I would never want to put the success of steemit inc or steem itself into the hands of one 3rd party. talk about centralization.

·
·
·
·
·

Steemit.com doesn't waste that much resources, it's the api that pretty much powers nearly the whole ecosystem.

  ·  7 months ago (edited)

A little skeptical of the performance impact of moving too much consensus data to rocksdb, but it is absolutely worth pursuing it via a cautious methodology that involves selectively moving data and careful regression testing as described in the post. At this point, the transaction performance seems hardly a bottleneck and if and when it becomes one that can be addressed at that point.

Good way to decentralize development and allow more priorities to be pursued at the same time rather than competing for the attention of one team.

Support.

·

I have this same concern. If we can match the performance I’m 100% in but I worry we won’t know until we try it (and it’s a lot of work) and we will find out it isn’t as optimal as the current solution but the current solution has major scaling issues and is forcing us to even more centralization due to costs.

·
·

We have done preliminary tests that made us comfortable with this solution. The account history rocksdb plugin was exactly one such test. We also wrote tests to verify rocksdb performance loading/unloading objects, etc and it left us reasonably comfortable that this approach will work. Of course there's some risk, but we're bearing that burden more than anyone else. If the final results are such that Steemit doesn't want to accept the code, we don't get paid.

·
·
·

I'm really excited to see Follow and Tags in Rocksdb, I'm all for it.

·
·
·

I think that making it more clear that if the end result isn't acceptable, then no fee is paid, will go a long way to getting naysayers behind this.

·
·
·
·

You might be right, but mostly there hasn't been many naysayers, so I thought it was clear enough.

So far as I can recall, it mostly seemed like one guy, and I tried to discuss it with him, but he seemed pretty set on his position so I didn't figure it was worth more effort.

·
·

At least with a match in performance, RocksDB has a rich ecosystem of tools for devs to then start playing around with different schema combos and see what performance can be gained that way. I suspect something might lie in that direction because the nature of storage for a blockchain is different to the storage that better supports API calls (e.g. .getDiscussions('trending', ...))

i have no idea what are you talking about but it sounds good :D
if it is doable, if it will reduce costs, and if you know you can do it, well it needs to be done.

·

no techy either. reduce costs increase revenue. what can i do to help. love this site

Loading...

Thank you for making this post and describing your proposal. Ned's post earlier wasn't exactly clear...or perhaps I was just distracted when reading it.

Whether or not your team does this, I think the best thing for the community is for development moving forward to incorporate a more open source community driven mentality. While Steem and Condenser and the majority of other projects involving Steem are open source...they're really rather closed considering. Most open source projects have forums and timelines and such. On Steem, they rarely even talk about the development. There are a TON of developers here. They can help. They could greatly improve Steem and reduce timelines for development. Right now they're just supposed to fork the code and push any changes they have I guess?

If Steem views this as financially beneficial to free their devs from other things, then they should do it. But...I hope if you have any devs with open source experience, that they help get the community involved as well.

I’m guessing that Ned isn’t going to accept this proposal...

https://steemit.com/steemit/@ned/rocksdb-and-smts-announcement

That’s too bad. I was really hoping that a competent dev team would take over some of the development around here. Ned has apparently learned absolutely nothing in 2.5 years. What a waste of money and potential.

·

Yep, they decided they wanted to do it themselves.

·
·

Well, at the very least you sparked a fire that needed to be lit. Thanks.

·
·

Yeah...and they have such a stellar track record of doing things themselves.

When do you think it’ll be time to look for a new blockchain “home?” Doesn’t seem that Ned & Co. are interested in collaborative development...or really any good/useful development. They’re still fixated on SMTs - and a lesser version of the original idea at that - while everything else around them burns.

They’re killing the potential of this blockchain.

·
·
·

I'm glad they passed on BT expensive deal.

·
·
·
  ·  7 months ago (edited)

It seems selling hopes and dreams is much easier than providing legitimate code. The Smoke Blockchain exists, and they innovated upon what steemit built over 2.5 years. At least with SBD and delegations removed, ram stays low enough the coin economically works in the future. How will new users power up when they know Ned is powering down? This is a fatal flaw. A flaw the witnesses of steem are powerless or too lazy to tackle. How do we encourage investment in, when the "owner" of 70% of the stake powers down? Steemit stagnated on it's innovation. Opening up doors for Smoke blockchain developers and coming dapps to try things like evergreen content. Could Steem even do evergreen? Doubt it. The users would flag it thinking it increases ram, when it actually reduces ram/spam at the same time. Doh.

·
·

That is not good. If cost was issue then that should have been negotiated. But for the sake of decentralised development this proposal should have been accepted to learn and open more into this direction

·
·

Oh well - You gave them a decent offer. With even less staff members do you feel the Steemit team can meet the same timeframe that you proposed?

What do you think of their proposed SMT-Lite solution?

·
·
·

I'm very skeptical on their rocksdb timeline. It looks like an estimate to just to write the code, without assuming adequate time for building a good test framework.

I've privately been advocating starting with SMT-Lite and rolling out more features later, so I'm happy with that change.

I'm not sure what the first couple pages mean. I read it a few times, and yet I'm still pretty confused. Sounds a bit like technobabblejargon. Is the tldr "things are getting more expensive"?
And if so, was this anticipated by those making the decisions?

·

tldr is: Memory requirements are increasing and will increase more as more users join the network and transact there (voting, funds transfers, etc). In order to keep costs under control as the network activity increases, we need to use disk storage instead of memory storage for this data. The rest is details about how to do it.

Yes, it was anticipated that these resource requirements would increase as activity on the blockchain increased. There was already one prior attempt a long time ago to fix this problem called "chainbase", but it didn't really solve the problem very well. Rocksdb is essentially an improved replacement for chainbase.

Loading...



This may be the greatest idea anyone on your primitive planet has ever had.

You have the full support of The Empire and its vast supply of resources.

Makes a lot of sense, I think. While it seems obvious at first, I'm a little confused about the request for the money, however.

Given that it's an open-source project, where blocktrades stands to benefit greatly from the improvement (both through, hopefully increased steem value, and reduced overhead costs) and is currently already a Top20 witness with a responsibility to maintain or otherwise assist in upgrading the blockchain-- I feel like the appropriate steps would be to gauge sentiment, and then proceed forward regardless of Steemit being onboard or not.

Wouldn't blocktrades benefit from the reduced overhead costs as well? Doesn't blocktrades already benefit substantially by being one of the go-to "Third Party Exchances" linked to directly on Steemit? I seem to recall discussion from numerous parties about how developing the blockchain SHOULD be a core requirement of a witness.

Don't get me wrong -- I think this should almost certainly be the path forward.

This should be done as well as the issue where images and video are stored? As i understand it is stored in some CDN server in the cloud. As a steem witness /node operator i donot want to deal with images or videos .Witness /node operator shall not be liable for content..(nothin happened so far, but with legal framework and its outlook, better be safe than sorry)

·

The actual images and video aren't stored on witness nodes, those are stored on separate servers, so witnesses are definitely not liable for such content.

·
·

This was not the consensus from team members a year ago. They were not sure who would be liable. But since its stored in separate server,its a relief

If I (technical noob) understand correct, this would allow witnesses to run cheaper nodes which would probably lead to more nodes or at least less nodes in danger of being turned off?!

If so, I think this is GREAT idea, especially for this cost.

·

Yes, your understanding of the proposal is correct.

I see a few of the witnesses I vote for answering, so that's good news.
Curious to hear from more.

Is there a plan for a large scale test?
I see you mention thoroughly testing but is it a large scale with real case data?
I seem to recall the lack of test db case on the last HF.... that bit us all.
Particularly hard hit were some who's accounts were smaller, etc. I know the data movement is nothing like the cahge in costs (RC, etc) but still,. without the large test with real world users and data, in a test net, you never know what you don't know

BUT, first pass, I love the concept!

·

Yes, to us thorough testing always means incorporating realworld data when it's possible (and it is in Steem's case). Our team has long been a proponent for more intensive testing of the blockchain code.

·
·

yeah, a SOLID test-net is important
Happy to hear

I think they should work together for good feedback.

that would be great not only for the nodes but also for the blockchain in general, besides that the users need the SMT that is expected to recover the interest of the investors improving the situation of Steem for the next year of which great things are expected in the crypto world

I am not a technical expert on the aspects of data storage and servers but I would say that this idea could signal a critical pivot point where the community and the witnesses that govern the protocol take more ownership to make the ecosystem more sustainable in the future. This is an excellent opportunity as it would be part of an established project that has been able to generate revenues from activities. I hope these ideas are further explored with Steemit to ensure steps are taken to establish a viable business model throughout the ecosystem. Thanks for sharing!

Posted using Partiko iOS

Technicalities are being discussed by competent people but I am all for the proposal for simple reason that it will start decentralizing the development teams. To me that that is most important. 250K is not a big cost to test such initiatives.

This is one of the best proposals I have seen. You have ran a successful POC and the price for a six team project is modest and of course you mentioned a regression test system for further work which I assume will be at least partly automated.

This is a win win for Steemians. Than you for proposing this. I hope it gets taken up.

I don’t have the technical to skills offer up a reasoned Aye or Nay but do recognize several commenters here who I think do have skills indicating Aye. For what it’s worth (0.0001 DOGE at current exchange rates) I wonder though if this runs counter to the idea of a move to more decentralization.

Posted using Partiko iOS

·

One of the key goals of this project is to decrease node costs to the point where it's cheap for anyone to run a node. I believe this will lead to greater decentralization. Currently most if not all of the Steem front-ends rely on Steemit servers, for example. I really feel this has to change.

·

... if this runs counter to the idea of a move to more decentralization.

No. The proposal encourages greater decentralisation. Right now it takes a lot of resources to run a seed node and so very few people do it. Moves, like this proposal, lower the bar for running servers and if the costs reduce enough there will be more public (and private) API nodes. Since API nodes are currently a major point of centralisation, having more will make Steem more decentralised.

I think this it is not an urgent issue to solve for the Steem blockchain although it would be nice to do it.

Rather you should put your experience and resources into solving SMT's.
I have the impression that Steemit Inc is stuck on that and your experienced team would be much more appreciated. It is also a great business opportunity, I'll bet that a lot of dapps teams would be interested into a faster solution even if they have to put more money on the table.

Thanks for the comprehensive post for noobs like me. It seems a good solution for Steemit.inc problems nowadays, so go for it!! :)

Posted using Partiko Android

Oh yes yes yes.

I'm supporting your approach and Steemit Inc should'nt be the only main contributor to steemd project.
And yes, searching solutions to lower infrastructure coast by using better solution is a good practice in IT world

This proposal is exactly what we need right now. Full support!

My steem witness vote for you. <3

I support this project!👏🙌👍😁 ✔🔝💯

Posted using Partiko Android

Chainbase was an early attempt to reduce the memory requirements of running a Steem node, but it ultimately failed because the storage mechanism was poorly tuned for storing data on a hard disk,

I was always curious about the performance of ChainBase as it claimed to similar to LevelDB and other DB engines. So information that you have provided answers some of it. All the best and will look forward to see the benchmarks.

I'm incompetent to comment on programming matters, but I do understand business opportunity and costs. I'm strongly in favor of reducing the expense necessary to implement nodes, as that increases the number of entities that can undertake to do so.

While increasing the number of teams developing on the blockchain is good, for the same reason that increasing the population of nodes is good, I'm too poorly informed as to how this proposal affects that. Mostly folks with specific expertise and skin in the game will have to evaluate this proposal and vote with their wallets, which is appropriate.

I hope this is the beginning of both less expensive and many more nodes, and many more devs working on the blockchain.

Thanks!

How profitable is it to be a witness?

·

Depends on the rank of the witness. We are #73, mine about 8 blocks each day, x by 30 = 240 SP. Latest bill, 310 STEEM.

You have my vote.
While others have suggested solutions, this is the first concrete one (I see) from someone who can actually practically implement it.

I hope Ned goes for it. It would be a confirmation that he actually still cares about Steemit. At Steemfest, sadly, I got the opposite impression.

awesome!

makes to me perfect sense. But i have now a question.

If we have in 2 years 20 times more users / more daps etc. And we get now the problems about the memory/ costs.

How would this work Out? How much more cost efficient can be a new Database Solution?

Because with SMTs i think we need more memory or i am wrong?

reduce costs is for the Network/ witnesses then super important right?

·

SMTs will certainly require more memory if they get market traction, as that will lead to more transactions on the blockchain. So, yes, I think this work is very important to the future success of SMTs and for platform growth in general.

Your site isn't working this morning. It shows that spinner for loading data then errors out 503.