Steemit Engineering Update: Cost Reductions, RocksDB/MIRA, Condenser Split

in #steem5 years ago (edited)

Engineering Update 2-6-19.jpg

Hello Steemians, we’ve been hard at work reducing the costs of running steemd nodes, adapting RocksDB for use in Steem, and splitting condenser into separate Wallet and Social applications. In today’s post we’d like to update you on the progress our engineers are making on these fronts. By sharing this information we hope to provide deeper insights into our operations, while also helping other node operators (like app developers) to reduce their costs as well.

EC2 Instance Cost Reduction

The full stack that Steemit runs is actually comprised of 14 individual pieces of software. Some of the more well known pieces are steemd, condenser, and hivemind but there are also lesser known applications like jussi, conveyor and even redeemer. Each of these pieces of software runs on their own EC2 instances, which can be thought of as cloud based servers.

The Goal: Optimization

The goal for this project is to optimize the types of instances we are using for each piece of software. To accomplish this we analyze metrics like CPU and RAM utilization over a specified period of time for a certain amount of traffic. By discovering which instances are being underutilized, we can achieve significant cost savings simply by switching to a lower cost instance type. After that we can look at the process through which instances are scaled up and down to handle the need for additional processing power. We can reduce costs even further by ensuring that the instances are no bigger than necessary given the computational power that is required at any particular moment to meet the demands of the network.

These changes must be made carefully so that overall response time stays relatively constant, while ensuring that scaling happens quickly enough to support the additional load that results from high-traffic periods.

Status

So far we have completed this task for four of our applications. We were able to reduce our average monthly expenditures for condenser (the software that powers steemit.com) by 60%. We were also able to reduce expenditures on conveyor by 75% while maintaining performance. Once this project is completed we will be able to turn our attention to whether or not we should utilize “reserved instances” which could deliver additional cost reductions in the range of 20-35% with respect to total EC2 spending.

RocksDB and MIRA

2 months ago we announced that we were developing a new piece of software we are calling "MIRA," which stands for Multi Index RocksDB Adapter. The purpose of MIRA is to allow Steem blockchain nodes to store almost all necessary data on disk in a modern database instead of in RAM. At that time our focus was on getting RocksDB working with data consistency.

We have now had a correct implementation of MIRA that passes all of our tests for several weeks and have progressed to focusing on performance optimizations. There are around 200 options we can use to tune RocksDB in order to maximize performance. We are continuing to optimize performance, but we have made enough progress to justify prioritizing the deployment of RocksDB to our development environment so that we can test live performance. Our goal is performance parity on average hardware, which will make running a steemd node much less expensive.

Condenser Split

Finally, we have made significant progress with respect to separating condenser into two separate applications, one social and the other financial. The process involved forking condenser and then carefully cutting whatever features do not belong in the given application. The goal is to have a complete separation of concerns. The financial version doesn’t need any of the social features, and the social version doesn’t need any of the financial features.

We have now eliminated all of the social features from the wallet app, and are in the process of streamlining the user-experience in the wallet application. On the social side, we are removing components which rely on active key as well as any signup API services (which are now part of the wallet service). To learn more about why we’re creating two separate condensers, check out our previous post.

Follow @steemitblog

There is a lot of work left to do, but we are extremely happy with the progress that has been made to date. We look forward to providing continued updates on this process, so be sure to follow @steemitblog if you would like to stay informed about the work we are doing to increase the sustainability and scalability of the Steem ecosystem.

The Steemit Team

Sort:  

Very impressive work so far on cost reduction. We all benefit from this in the end.
The test of a company, community or ecosystem is how it performs in times of crisis and Steem and Steemit have performed admirably where it counts.

not we should utilize “reserved instances” which could deliver additional cost reductions in the range of 20-35% with respect to total EC2 spending.

Reserve instances is a good idea. I am actually surprised that we are not using reserved instances yet.

Regarding Wallet separation: please add some information and links about Steem wallets on steem.com, because this important information is still missing on the website. Users and investors would like to know how to save, transfer and stake Steem, when they visit Steem.com.

Thanks for the suggestion!

With a bit of effort, the "social" aspects of Steem can still save themselves and grow, I believe.

And it matters, because whereas it's lovely to market Steem as "a great place to develop dapps," it's even better to market Steem as "a great place to develop dapps" that also happens to have a huge and thriving built-in user base.

You guys did a great job by removing our views, so that we authors have no more a clue about how many people watch our posts. Why not continue this philosophy on the images? You already made a step in the right direction by reducing image quality to 1% of the original, but why not go the whole way? By completely removing images you would save even more costs!
I already use https://postimages.org/ when I want to present high quality images to the 2 or 3 voters that come by my posts. It's completely free. And you don't have to power up. So for authors with small accounts with little RC, this would be a nice solution.
Just my 5 steems.

Just fyi don't confuse with https://www.imagepost.com/

And don't click on this last link...

1486484443_pig.gif

You did? I warned you!

Now that Hivemind is live it will actually be easier for us to add back view counts. View counts are extremely easy to game, but with Hivemind we could make adjustments much faster. At some point we do hope to get to that feature, we're just very busy now focusing on our priorities. But thanks for the feedback!

Nice to hear somebody isn't sleeping up there. But don't wait to make this place liveable too long. According to @arcange 's data, this platform has only 2-3 months left before it's dead:
https://steemit.com/steemit/@pagandance/steemit-death-spiral-on-track

If I understand him correctly, @kingscrown has resorted to a desperate post like this to see what feedback/impact he has towards his 'audience', by lack of views:
https://steemit.com/steemit/@kingscrown/test-if-you-read-this-post-please-upvote-and-or-reply

If you are unhappy about the pic size on steemit, why don't you use steempeak as an inteface? Or be creative and use postimages.org. Which you do, and a small rant is always fun.
Are you from Vienna?

Let's not make things more complex than they already are. And no I'm not from Vienna, but considering moving east of the border over there.

Sounds very complicated way above my understanding level of how websites/blockchain work. Keep up the good work and hopefully price rises in the future can justify the work you are doing as well as whatever steemit is able to pay you.

Any report on the SMTs ?

I might be doing a video to help explain just an quick overview of all the major changes happening with steem, do you think Mira and the new condenser news along with RocksDB and Hivem,mnd and hoping to put Full Nodes on regular 4gb Servers, do you think that is a good start for the system upgrades? I also plan to go over Worker proposals, Steemit Ads, and a few other things

Good work. The community will always support this development.

Posted using Partiko iOS

Add more features to the social side of the condenser, analytics is one of the key features missing in the app.

Posted using Partiko Android

It's going to get way easier to add features to steemit.com once the wallet features are separated out into a different app.

Coin Marketplace

STEEM 0.23
TRX 0.12
JST 0.029
BTC 66649.32
ETH 3562.88
USDT 1.00
SBD 3.12