Steem Blockchain Update August 2017

in #steemitdev7 years ago (edited)

The last few months have been busy for our blockchain team. We are making great progress and want to share what we have been doing with you.

HF 20

A little over a month ago we made the tough decision to postpone HF 20 implementation in favor of enhancing our application framework. We discussed this decision with the witnesses and they were supportive of this change. We are still planning on implementing and releasing HF 20 in the coming months.

Scaling the Blockchain

In the meantime, we have been working on some fundamental changes to help scale Steem. We currently run 7 steemd nodes in AWS to support steemit.com and will need to continue to run more and more to support our growth. The key problem is that Steemd is single threaded, however, we are well along in the process of making it multithreaded.

Imagine the steemd server program is a stove. Right now, it has a single burner and we need to make a meal. A few of the dishes are the peer to peer network, accepting and evaluating blocks, and responding to API requests for data clients like steemit.com. We are rapidly switching pots, pans, and skillets onto the single burner to make each component of the meal. The increased request volume from steemit.com is delaying the meal.

Thus far, we have been buying extra single burner stoves to distribute the load because this was how blockchains were originally designed. But this is not ideal because all of our stoves (our computers) actually have 8 burners (CPU cores) and we are only using 1 each. We have been making changes to the steemd server program so that we can use all 8 burners and not need as many stoves. That way we can put a skillet on one burner, a pot on another, pan on another, you get the idea.

DQmUTfpJduJ7hFmcQokRhY1YKb6tsxdkPnt991rmaMDa8tm_1680x8400.png

Previously, we had a single application that handled the p2p code, the database, plugins, and APIs. Now, everything is a plugin and only needs to communicate with other plugins that they directly depend on. This increased level of modularity allows for quicker development, easier code review, and greater parallelism. It will also allows us to switch components in and out without breaking the greater whole. We are now developing with building blocks rather than specifically designed puzzle pieces.

FABRIC

Our early tests are showing a 5-fold increase in throughput for servicing API requests. This refactor also lays the foundation to scale steemd into the future and implement the Steem Blockchain Fabric.

There are some API changes on the steemd node that should make accessing data in the blockchain easier and allow upgrading of our APIs without breaking existing calls. We have added a condenser_api which exposes all existing APIs so that we do not break backwards compatibility with existing apps. We have also rewritten the jsonrpc parser and it is completely jsonrpc 2.0 compliant. We will announce when this is deployed to staging.

Most of the work we have been doing is highly technical and challenging, but we look forward to keeping the community abreast of our work and strategies, on much of which we look to the Witnesses and community for collaboration. We’re happy to announce that we are confident these solutions will provide a robust foundation for meeting all of our future scaling needs and enable us to turn our attention to the features you all want to see.

Team Steemit

Sort:  

I am a layman when it comes to understanding the depths of blockchain tech. Although I am interested in the overall application to Steemit since I am an investor in Steem and the platform of Steemit.

I had seen posts speaking about scaling issues that would be faced so it was nice to read about what you are working on in a format that is easily understood by me. Thank you!

@overkillcoin and I updated the graphic for you to use! Link to Post

Nice format you got. Was much easy to understand with your format.

Courtesy of Interition Ltd

Nice.. Thanks!

Hi is this the steemit networks work presentation on network
really eye opening I Love Steemit
as years back we where on MLM network but
we are now smart now we are on blockchain network

Steemit

Love

U

Courtesy of Interition Ltd Gource

FTFY. Credit where credit's due.

I don't have the technical knowledge to understand it :) but glad it was useful for someone. That's what we were hoping for when we made it so thank you for letting us know and we appreciate the upvote as well!

Also the version in the blog post link should view better than this one as well if needed.

Yeah its very good

Speaking of layman's problems is not something to talk about.
The important thing in knowing is who wants then he will get it. So you have done something amazing that can improve the mindset of others and the spirit that can make steemit always in the hearts of the layman and the professional.

a very confusing statement indeed

Why did you confuse

"The important thing in knowing is who wants then he will get it."
I can't make heads or tails of that

Hahaha, what is your mean ??? Whats you want me to

This sentence is also incomprehensible, I'm sorry I don't want to be a grammar nazi, but Mabey use a translation program.

Its okay dear. Use the language that easy understand by others people. Thanks for recommendation. I appreciate it, next time I will do it. Cheerss.

impressive content! No doubt that's great!

hmmm.... i couldn't tell if you were being rude or making conversation... lol. I'll get to the bottom of it! Following :)

Haha lol. How about you if we have a personal room to chat ??

Well, now that you mention it! I do want to find some more people to interview on my tv show, we could chat via Zoom! Let me know if you would be interested! We could talk about your sense of humor ;) or your travels and art ;)

Great thats amazing, interesting words to read. So, whats I will have from the interview LOL ?
I am funny guy. I can make every girl falling in love with my tooth. Haha.

Thanks for that, that's great! The post has been updated.

Wow! really love the community here, and to see everyone working together! Thanks for the love we could witness @steemitadventure @justinw and @overkillcoin <3

Thank you @jacquelyne. Even though I'm relatively new here, it's great to be a very small part of the incredible effort and talent involved in Steemit. I'm sure @steemitadventure would agree.

Glad you liked and used it! If you ever come across anything you think would help Steemit feel free to let me know. We work on everything from infographics to explainer videos etc.

We have been pondering a few things but as of now lack the technical (blockchain) knowledge to compose it well.

Happy to assist :)

idk

Safe to say I am a visual person, and its nice sometimes to have little pictures summarizing something up like that! Thanks

I am the same way, probably why I illustrate! You are very welcome and glad you found it useful.

Yes its easy get well lots of thanks to jerrybanfield

looks good!upvoted!

Thanks for the impressive content, No doubt that's great!

Oh really, you can also check my blog updates
https://steemit.com/@tsandesh

hehehe you said abreast hehehehe

Hello @trafalgar! I am sorry to intrude like this on a comment of yours in another post. I saw you are voting sometimes for @ocd posts. I happen to be one of the curators working with acidyo and its team.

Recently I made a post where I talk about my story on steemit in order to participate in blueorgy´s Steefmest ticket giveaway, do you mind checking it out? Perhaps you like it and would like to vote on it, I am going to use all the rewards from that post to buy a plane ticket to Lisbon :)

This is the post and again, sorry to intrude here :) Thank you for your time!

hehehe... thx for your support on my article today. love steemit more by the day..

LOL, at your comment man.

--

It is late for me, but just stopping by again to say thanks for a vote and the support -- you have been a big help to me and I appreciate it every time.

u still alive?

WOW $22 on one comment! dude come back! where did u go! this was 2 months ago! we need you back! You make SO much on ur posts its so awesome!

I saw in an old post of mine 4 months ago you upvoted me and helped me make one of my higher earning posts of all times and i really really am grateful for this thank you so much! I know for you it was just one click but It meant a lot! Thank you! I will try to posty more about my ayahuasca plants and other tropical plants of mine!!

I think the right decision was made choosing blockchain performance over ease of unboarding. Both are very important, but there is no point in a ton of users if blockchain can't handle it.

Very glad that core team still developing blockchain. This means that STEEM alive a well. There is silence sometimes from devs and it is frightening a bit.

Imagine the steemd server program is a stove. Right now, it has a single burner and we need to make a meal. A few of the dishes are the peer to peer network, accepting and evaluating blocks, and responding to API requests for data clients like steemit.com. We are rapidly switching pots, pans, and skillets onto the single burner to make each component of the meal. The increased request volume from steemit.com is delaying the meal.

Would EOS not be the ultimate solution for scaling steem?

Thank you for the update.. keep on trucking

Awesome, multithreading is definitely a huge key to scaling. We will be well prepared for hundreds of millions of users soon.

This really is great news. You are tackling the issues which have been noticed lately as occasional hiccups of the frontend.

Great approach of making the backend software multithreaded and divided into individual dedicated workers.

If I assume correctly this development is not changing the structure of the database itself, isn't it? Maybe, sometime in the future, when even multithreaded approach won't suffice, a quantum leap into sharded blockchain will be necessary.

Are you thinking about this possibility already or is it too far in the future.

A side question - does this breaking of the code into plugins and apis means that community based nodes will be possible serving specific plugins only.

Thanks for your work. And thanks for the answers! Sorry if you'll find my questions badly formulated. I am not a backend / blockchain / daemon / node programmer :)

If I assume correctly this development is not changing the structure of the database itself, isn't it? Maybe, sometime in the future, when even multithreaded approach won't suffice, a quantum leap into sharded blockchain will be necessary.

That's something we've been planning for a while - the fabric architecture. This is a step toward that future.

A side question - does this breaking of the code into plugins and apis means that community based nodes will be possible serving specific plugins only.

That architecture is already in place now, and different types of users can run different sets of plugins depending on their needs. :)

Great. Thanks a lot for your reply. Now all I need is a set of simple instructions, a tutorial, on how to put up a node. Hardware and software requirements, setup, options, maintenance, ... Basic stuff :)

Maybe I've missed it and it's already out there? I mean a recent one. Heck, there should be an officially sanctioned tutorial someplace on steem.io site.

I remember this one was the post right before I started to buy steemit. Flash forward 6 months and we already are getting results in real code!

Good post and good news.

So I suppose it will now be easier to run a node on a low RAM server by not turning on all the plugins?

Is there any solutions being thought of to minimize the RAM required for a node with all calls enabled, i.e. with the condenser_api plugin enabled?

So I suppose it will now be easier to run a node on a low RAM server by not turning on all the plugins?

That's already possible today. :)

Is there any solutions being thought of to minimize the RAM required for a node with all calls enabled, i.e. with the condenser_api plugin enabled?

Those apis require a lot of fast i/o to the state database.

You have two options: you can put the database onto a ramdisk, or you can put it onto an extremely fast storage array. We use RAM in production because it's fastest, but if you have fast enough disks, you can get away with using a lot less RAM.

Considering to run a node with a ramdisk you mentioned. Need to upgrade RAM to 64, maybe 128Gb. Internet Connection 100Mbit.
From my understanding running a node is only rewarded when you are a witness. Right?

Awesomeness!!👍👍👍

cute graphic!! lol thanks for the fun vibes @karenmckersie

Glad you liked it !! Your very welcome!!!

HI Karen! Would you be interested in being interviewed on my tv show via Zoom? Would love to have someone knowledgeable about crypto currency come on! Let me know if you're open to it!

Oh no haha !! Im know very little about crypto !! Im just learning like a lot of us ! 👌✌👍😂

WoW Karenmckersie I like it its on Speed>>>

Haha thanks that was so long ago !!😂💕👌✌👍

Coin Marketplace

STEEM 0.29
TRX 0.12
JST 0.036
BTC 65930.92
ETH 3387.67
USDT 1.00
SBD 4.75