Steemit Engineering Update: Cost Reductions, RocksDB/MIRA, Condenser Split

in steem •  15 days ago

Engineering Update 2-6-19.jpg

Hello Steemians, we’ve been hard at work reducing the costs of running steemd nodes, adapting RocksDB for use in Steem, and splitting condenser into separate Wallet and Social applications. In today’s post we’d like to update you on the progress our engineers are making on these fronts. By sharing this information we hope to provide deeper insights into our operations, while also helping other node operators (like app developers) to reduce their costs as well.

EC2 Instance Cost Reduction

The full stack that Steemit runs is actually comprised of 14 individual pieces of software. Some of the more well known pieces are steemd, condenser, and hivemind but there are also lesser known applications like jussi, conveyor and even redeemer. Each of these pieces of software runs on their own EC2 instances, which can be thought of as cloud based servers.

The Goal: Optimization

The goal for this project is to optimize the types of instances we are using for each piece of software. To accomplish this we analyze metrics like CPU and RAM utilization over a specified period of time for a certain amount of traffic. By discovering which instances are being underutilized, we can achieve significant cost savings simply by switching to a lower cost instance type. After that we can look at the process through which instances are scaled up and down to handle the need for additional processing power. We can reduce costs even further by ensuring that the instances are no bigger than necessary given the computational power that is required at any particular moment to meet the demands of the network.

These changes must be made carefully so that overall response time stays relatively constant, while ensuring that scaling happens quickly enough to support the additional load that results from high-traffic periods.

Status

So far we have completed this task for four of our applications. We were able to reduce our average monthly expenditures for condenser (the software that powers steemit.com) by 60%. We were also able to reduce expenditures on conveyor by 75% while maintaining performance. Once this project is completed we will be able to turn our attention to whether or not we should utilize “reserved instances” which could deliver additional cost reductions in the range of 20-35% with respect to total EC2 spending.

RocksDB and MIRA

2 months ago we announced that we were developing a new piece of software we are calling "MIRA," which stands for Multi Index RocksDB Adapter. The purpose of MIRA is to allow Steem blockchain nodes to store almost all necessary data on disk in a modern database instead of in RAM. At that time our focus was on getting RocksDB working with data consistency.

We have now had a correct implementation of MIRA that passes all of our tests for several weeks and have progressed to focusing on performance optimizations. There are around 200 options we can use to tune RocksDB in order to maximize performance. We are continuing to optimize performance, but we have made enough progress to justify prioritizing the deployment of RocksDB to our development environment so that we can test live performance. Our goal is performance parity on average hardware, which will make running a steemd node much less expensive.

Condenser Split

Finally, we have made significant progress with respect to separating condenser into two separate applications, one social and the other financial. The process involved forking condenser and then carefully cutting whatever features do not belong in the given application. The goal is to have a complete separation of concerns. The financial version doesn’t need any of the social features, and the social version doesn’t need any of the financial features.

We have now eliminated all of the social features from the wallet app, and are in the process of streamlining the user-experience in the wallet application. On the social side, we are removing components which rely on active key as well as any signup API services (which are now part of the wallet service). To learn more about why we’re creating two separate condensers, check out our previous post.

Follow @steemitblog

There is a lot of work left to do, but we are extremely happy with the progress that has been made to date. We look forward to providing continued updates on this process, so be sure to follow @steemitblog if you would like to stay informed about the work we are doing to increase the sustainability and scalability of the Steem ecosystem.

The Steemit Team

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Very impressive work so far on cost reduction. We all benefit from this in the end.
The test of a company, community or ecosystem is how it performs in times of crisis and Steem and Steemit have performed admirably where it counts.

not we should utilize “reserved instances” which could deliver additional cost reductions in the range of 20-35% with respect to total EC2 spending.

Reserve instances is a good idea. I am actually surprised that we are not using reserved instances yet.

Regarding Wallet separation: please add some information and links about Steem wallets on steem.com, because this important information is still missing on the website. Users and investors would like to know how to save, transfer and stake Steem, when they visit Steem.com.

·

Thanks for the suggestion!

With a bit of effort, the "social" aspects of Steem can still save themselves and grow, I believe.

And it matters, because whereas it's lovely to market Steem as "a great place to develop dapps," it's even better to market Steem as "a great place to develop dapps" that also happens to have a huge and thriving built-in user base.

You guys did a great job by removing our views, so that we authors have no more a clue about how many people watch our posts. Why not continue this philosophy on the images? You already made a step in the right direction by reducing image quality to 1% of the original, but why not go the whole way? By completely removing images you would save even more costs!
I already use https://postimages.org/ when I want to present high quality images to the 2 or 3 voters that come by my posts. It's completely free. And you don't have to power up. So for authors with small accounts with little RC, this would be a nice solution.
Just my 5 steems.

Just fyi don't confuse with https://www.imagepost.com/

And don't click on this last link...

1486484443_pig.gif

You did? I warned you!

·

Now that Hivemind is live it will actually be easier for us to add back view counts. View counts are extremely easy to game, but with Hivemind we could make adjustments much faster. At some point we do hope to get to that feature, we're just very busy now focusing on our priorities. But thanks for the feedback!

·
·

Nice to hear somebody isn't sleeping up there. But don't wait to make this place liveable too long. According to @arcange 's data, this platform has only 2-3 months left before it's dead:
https://steemit.com/steemit/@pagandance/steemit-death-spiral-on-track

·
·

If I understand him correctly, @kingscrown has resorted to a desperate post like this to see what feedback/impact he has towards his 'audience', by lack of views:
https://steemit.com/steemit/@kingscrown/test-if-you-read-this-post-please-upvote-and-or-reply

·

If you are unhappy about the pic size on steemit, why don't you use steempeak as an inteface? Or be creative and use postimages.org. Which you do, and a small rant is always fun.
Are you from Vienna?

·
·

Let's not make things more complex than they already are. And no I'm not from Vienna, but considering moving east of the border over there.

Sounds very complicated way above my understanding level of how websites/blockchain work. Keep up the good work and hopefully price rises in the future can justify the work you are doing as well as whatever steemit is able to pay you.

Any report on the SMTs ?

WOW... deep Thx for that 🤞✌

Good work. The community will always support this development.

Posted using Partiko iOS

Add more features to the social side of the condenser, analytics is one of the key features missing in the app.

Posted using Partiko Android

·

It's going to get way easier to add features to steemit.com once the wallet features are separated out into a different app.

Another great update!
Oh yeah!

Looks like you are on good track. The community is very happy to see that and hear from you with the regular updates. Keep up the good work!

Why not investigating another cheaper cloud hosting like Vultr instead of AWS?

This is a great announcement and I hate to be "that guy" who comes in and takes a dump on the good vibes, but what about Smart Media Tokens?

I know, SMT's were scaled back and we are going to be getting SMT-lite while other features come later, but the silence is deafening. I also recall RocksDB being important to the launch of SMT's and now that RocksDB appears to be going well, how about SMT's?

In the interim, @yabapmatt @aggroed and @harpagon have launched an impressive side-chain called Steem Engine and while it is only basic for now, it gives us SMT like functionality on Steem right now.

The Steem community has always impressed me, but of late, even more so and it feels like they have been picking up the slack a little bit. We have all been feeling the pinch of this bear market, but still.

The increased transparency and communication lately has also been welcomed and appreciated, I can see STINC is making more of an effort now, but the silence about the pending mainnet launch in a month concerns me.

If you were concerned about cost why would you be using the MOST EXPENSIVE hosting service? :D Don't you think it would be cheaper to simply have the community deploy RPCs? Sunset steemit.com's centralized bullshit? Make truely decentralized apps? OR just waste your money on users who give you jack shit. I guess steemit.com wants that control over it's users just like facebook. As they already openly lied and censored user accounts , (even worse than YouTube or FB, they tell you when users are censored on the front ends)

Satoshi told us about this problem 9 frigging years ago. Remember the guy who inspired Ned and Dan's code? He is gone but his insight remains.
'Screenshot_2019-02-12_11-43-14.png

So great progress. Sounds really good to read that. Let's go forward. ;D