Steem Blockchain Update August 2017

in steemitdev •  last year

The last few months have been busy for our blockchain team. We are making great progress and want to share what we have been doing with you.

HF 20

A little over a month ago we made the tough decision to postpone HF 20 implementation in favor of enhancing our application framework. We discussed this decision with the witnesses and they were supportive of this change. We are still planning on implementing and releasing HF 20 in the coming months.

Scaling the Blockchain

In the meantime, we have been working on some fundamental changes to help scale Steem. We currently run 7 steemd nodes in AWS to support steemit.com and will need to continue to run more and more to support our growth. The key problem is that Steemd is single threaded, however, we are well along in the process of making it multithreaded.

Imagine the steemd server program is a stove. Right now, it has a single burner and we need to make a meal. A few of the dishes are the peer to peer network, accepting and evaluating blocks, and responding to API requests for data clients like steemit.com. We are rapidly switching pots, pans, and skillets onto the single burner to make each component of the meal. The increased request volume from steemit.com is delaying the meal.

Thus far, we have been buying extra single burner stoves to distribute the load because this was how blockchains were originally designed. But this is not ideal because all of our stoves (our computers) actually have 8 burners (CPU cores) and we are only using 1 each. We have been making changes to the steemd server program so that we can use all 8 burners and not need as many stoves. That way we can put a skillet on one burner, a pot on another, pan on another, you get the idea.

DQmUTfpJduJ7hFmcQokRhY1YKb6tsxdkPnt991rmaMDa8tm_1680x8400.png

Previously, we had a single application that handled the p2p code, the database, plugins, and APIs. Now, everything is a plugin and only needs to communicate with other plugins that they directly depend on. This increased level of modularity allows for quicker development, easier code review, and greater parallelism. It will also allows us to switch components in and out without breaking the greater whole. We are now developing with building blocks rather than specifically designed puzzle pieces.

FABRIC

Our early tests are showing a 5-fold increase in throughput for servicing API requests. This refactor also lays the foundation to scale steemd into the future and implement the Steem Blockchain Fabric.

There are some API changes on the steemd node that should make accessing data in the blockchain easier and allow upgrading of our APIs without breaking existing calls. We have added a condenser_api which exposes all existing APIs so that we do not break backwards compatibility with existing apps. We have also rewritten the jsonrpc parser and it is completely jsonrpc 2.0 compliant. We will announce when this is deployed to staging.

Most of the work we have been doing is highly technical and challenging, but we look forward to keeping the community abreast of our work and strategies, on much of which we look to the Witnesses and community for collaboration. We’re happy to announce that we are confident these solutions will provide a robust foundation for meeting all of our future scaling needs and enable us to turn our attention to the features you all want to see.

Team Steemit

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

I am a layman when it comes to understanding the depths of blockchain tech. Although I am interested in the overall application to Steemit since I am an investor in Steem and the platform of Steemit.

I had seen posts speaking about scaling issues that would be faced so it was nice to read about what you are working on in a format that is easily understood by me. Thank you!

@overkillcoin and I updated the graphic for you to use! Link to Post

·

Nice format you got. Was much easy to understand with your format.

·
·
·
·
·
·
·
·

Hi is this the steemit networks work presentation on network
really eye opening I Love Steemit
as years back we where on MLM network but
we are now smart now we are on blockchain network

Steemit

Love

U

·
·
·

Courtesy of Interition Ltd Gource

FTFY. Credit where credit's due.

·
·

I don't have the technical knowledge to understand it :) but glad it was useful for someone. That's what we were hoping for when we made it so thank you for letting us know and we appreciate the upvote as well!

Also the version in the blog post link should view better than this one as well if needed.

·
·

Yeah its very good

·

Speaking of layman's problems is not something to talk about.
The important thing in knowing is who wants then he will get it. So you have done something amazing that can improve the mindset of others and the spirit that can make steemit always in the hearts of the layman and the professional.

·
·

hmmm.... i couldn't tell if you were being rude or making conversation... lol. I'll get to the bottom of it! Following :)

·
·
·

Haha lol. How about you if we have a personal room to chat ??

·
·
·
·

Well, now that you mention it! I do want to find some more people to interview on my tv show, we could chat via Zoom! Let me know if you would be interested! We could talk about your sense of humor ;) or your travels and art ;)

·
·
·
·
·

Great thats amazing, interesting words to read. So, whats I will have from the interview LOL ?
I am funny guy. I can make every girl falling in love with my tooth. Haha.

·
·

a very confusing statement indeed

·
·
·

Why did you confuse

·
·
·
·

"The important thing in knowing is who wants then he will get it."
I can't make heads or tails of that

·
·
·
·
·

Hahaha, what is your mean ??? Whats you want me to

·
·
·
·
·
·

This sentence is also incomprehensible, I'm sorry I don't want to be a grammar nazi, but Mabey use a translation program.

·
·
·
·
·
·
·

Its okay dear. Use the language that easy understand by others people. Thanks for recommendation. I appreciate it, next time I will do it. Cheerss.

·
·

impressive content! No doubt that's great!

·
·
·

Of cour

·

Thanks for that, that's great! The post has been updated.

·
·

Wow! really love the community here, and to see everyone working together! Thanks for the love we could witness @steemitadventure @justinw and @overkillcoin <3

·
·
·

Thank you @jacquelyne. Even though I'm relatively new here, it's great to be a very small part of the incredible effort and talent involved in Steemit. I'm sure @steemitadventure would agree.

·
·

Glad you liked and used it! If you ever come across anything you think would help Steemit feel free to let me know. We work on everything from infographics to explainer videos etc.

We have been pondering a few things but as of now lack the technical (blockchain) knowledge to compose it well.

·
·
·

Happy to assist :)

·

idk

·

very good news ..

·

looks good!upvoted!

·

Thanks for the impressive content, No doubt that's great!

·

Yes its easy get well lots of thanks to jerrybanfield

·

Oh really, you can also check my blog updates
https://steemit.com/@tsandesh

·

Safe to say I am a visual person, and its nice sometimes to have little pictures summarizing something up like that! Thanks

·
·

I am the same way, probably why I illustrate! You are very welcome and glad you found it useful.

hehehe you said abreast hehehehe

·

Hello @trafalgar! I am sorry to intrude like this on a comment of yours in another post. I saw you are voting sometimes for @ocd posts. I happen to be one of the curators working with acidyo and its team.

Recently I made a post where I talk about my story on steemit in order to participate in blueorgy´s Steefmest ticket giveaway, do you mind checking it out? Perhaps you like it and would like to vote on it, I am going to use all the rewards from that post to buy a plane ticket to Lisbon :)

This is the post and again, sorry to intrude here :) Thank you for your time!

·

hehehe... thx for your support on my article today. love steemit more by the day..

·

LOL, at your comment man.

--

It is late for me, but just stopping by again to say thanks for a vote and the support -- you have been a big help to me and I appreciate it every time.

·

u still alive?

·

WOW $22 on one comment! dude come back! where did u go! this was 2 months ago! we need you back! You make SO much on ur posts its so awesome!

I saw in an old post of mine 4 months ago you upvoted me and helped me make one of my higher earning posts of all times and i really really am grateful for this thank you so much! I know for you it was just one click but It meant a lot! Thank you! I will try to posty more about my ayahuasca plants and other tropical plants of mine!!

I think the right decision was made choosing blockchain performance over ease of unboarding. Both are very important, but there is no point in a ton of users if blockchain can't handle it.

Very glad that core team still developing blockchain. This means that STEEM alive a well. There is silence sometimes from devs and it is frightening a bit.

Imagine the steemd server program is a stove. Right now, it has a single burner and we need to make a meal. A few of the dishes are the peer to peer network, accepting and evaluating blocks, and responding to API requests for data clients like steemit.com. We are rapidly switching pots, pans, and skillets onto the single burner to make each component of the meal. The increased request volume from steemit.com is delaying the meal.

Would EOS not be the ultimate solution for scaling steem?

Thank you for the update.. keep on trucking

Awesome, multithreading is definitely a huge key to scaling. We will be well prepared for hundreds of millions of users soon.

This really is great news. You are tackling the issues which have been noticed lately as occasional hiccups of the frontend.

Great approach of making the backend software multithreaded and divided into individual dedicated workers.

If I assume correctly this development is not changing the structure of the database itself, isn't it? Maybe, sometime in the future, when even multithreaded approach won't suffice, a quantum leap into sharded blockchain will be necessary.

Are you thinking about this possibility already or is it too far in the future.

A side question - does this breaking of the code into plugins and apis means that community based nodes will be possible serving specific plugins only.

Thanks for your work. And thanks for the answers! Sorry if you'll find my questions badly formulated. I am not a backend / blockchain / daemon / node programmer :)

·

If I assume correctly this development is not changing the structure of the database itself, isn't it? Maybe, sometime in the future, when even multithreaded approach won't suffice, a quantum leap into sharded blockchain will be necessary.

That's something we've been planning for a while - the fabric architecture. This is a step toward that future.

A side question - does this breaking of the code into plugins and apis means that community based nodes will be possible serving specific plugins only.

That architecture is already in place now, and different types of users can run different sets of plugins depending on their needs. :)

·
·

Great. Thanks a lot for your reply. Now all I need is a set of simple instructions, a tutorial, on how to put up a node. Hardware and software requirements, setup, options, maintenance, ... Basic stuff :)

Maybe I've missed it and it's already out there? I mean a recent one. Heck, there should be an officially sanctioned tutorial someplace on steem.io site.

I remember this one was the post right before I started to buy steemit. Flash forward 6 months and we already are getting results in real code!

Thankyou for the update, please keep them coming - this sort of direct communication from Team Steemit is needed especially when multiple influential members are spreading doubt about Steem's scalability..

·

If there has been one constant complaint, it has been about communication. The @steemitdev team is in a tough spot. If they focus on development it doesn't much leave time for communication. When they want to communicate it is never enough. Like you said, this sort of direct communication is vital and very welcome.

Good post and good news.

So I suppose it will now be easier to run a node on a low RAM server by not turning on all the plugins?

Is there any solutions being thought of to minimize the RAM required for a node with all calls enabled, i.e. with the condenser_api plugin enabled?

·

So I suppose it will now be easier to run a node on a low RAM server by not turning on all the plugins?

That's already possible today. :)

Is there any solutions being thought of to minimize the RAM required for a node with all calls enabled, i.e. with the condenser_api plugin enabled?

Those apis require a lot of fast i/o to the state database.

You have two options: you can put the database onto a ramdisk, or you can put it onto an extremely fast storage array. We use RAM in production because it's fastest, but if you have fast enough disks, you can get away with using a lot less RAM.

·
·

Considering to run a node with a ramdisk you mentioned. Need to upgrade RAM to 64, maybe 128Gb. Internet Connection 100Mbit.
From my understanding running a node is only rewarded when you are a witness. Right?

Awesomeness!!👍👍👍

·

cool graphic

·
·
·

Thanks ! im glad you liked it!

·

cute graphic!! lol thanks for the fun vibes @karenmckersie

·
·

Glad you liked it !! Your very welcome!!!

·
·
·

HI Karen! Would you be interested in being interviewed on my tv show via Zoom? Would love to have someone knowledgeable about crypto currency come on! Let me know if you're open to it!

·
·
·
·

Oh no haha !! Im know very little about crypto !! Im just learning like a lot of us ! 👌✌👍😂

·

WoW Karenmckersie I like it its on Speed>>>

·
·

Haha thanks that was so long ago !!😂💕👌✌👍

Thanks for the update! As a non-coder the analogy made it pretty easy to understand. ^^

·

Yes as I didn't have to sit here scratching my head ya know

Stove and making meals, you guys really like the "steem" theme I see :p. Keep up the good work guys, I trust your team is working their asses off.

Sounds great! Love forward progress

Cool dude!!

With the various changes that have been done no other just to simplify the user and facilitate the development team in the face of the problems encountered. With the addition of the stove it will add visitors to enjoy the dish. Thanks for information sir.

·

Dood.. are you a bot?

·
·

I am notva bot. I am alive like you too. Haha

·
·
·

Ok LOL

I have seen bots that crafted replies based on posts before, and the way your comment picked bits of the post and crafted the body of text, well, it had me wondering =p

·
·
·
·

Haha. So whre are u come from

·
·
·
·
·

I was raised on an island in Alaska, and escaped into the hellish jungles of Oregon, and effete civilization.

And you?

·
·

Iam not a bot. Why you ask me that question ?

·
·
·

I was mistaken.

While I don't understand every single thing in the post(especially the bit about fabric) I certainly am very much excited that sterns is becoming multi-threaded. This platform is gaining ~10k users every week, and that kind of growth will absolutely need sofrware that runs on multiple threads and multiple hosts.

Also, very happy to see that you have put in the effort to maintain backwards compatibility! This is something that companies that have a lot more resources than you guys don't do!

nice

Thanks for such an entertaining analogy. It made the more technical jargon easier to grasp. I'm confident that Steemit will only get better and better as time progresses. Can't wait to see what we'll be "served" next!

THank you for the update, hope to see a nice price increase due to this!

When will we see the next hardfork?

Great post!

·

How do you do it? That's such an original and pithy comment. Makes me wanna learn linux!

Fascinating. I'm sure going from single-threaded to multi-threaded for steemd is no easy task. But from the sounds of it, you guys are doing great. We sincerely appreciate all your hard work @steemitdev ! :)

I'm still concerned that the interface is worse than geocities, the site's growth is capped and users will continue leaving until that's addressed.

Good job guys, thanks for the update!

"The fabric is coming!" :)

Great to hear about and look forward to the continued improvements.
The multi-thread-ness will be a great improvement and ability to scale more easily.
Keep up the great work!

Thanks!

Thank you very much @steemitdev ,It is huge what you are doing to help steemit to be more performing in the future .
We must also applaud this courageous initiative on your part. Sometimes one is obliged to approach subjects a little critical.

il faut aussi applaudir cette initiative courageuse de votre part. parfois on est obligé d'aborder des sujets un peu critiques.

This is encouraging in the medium-term.

Which AWS plan do you run your full RPC nodes on please?

I see it reduces the replay time massively, but what is the RAM usage difference between SKIP_BY_TX_ID=[OFF/ON] when compiling steem?

Thanks for the communication.

I believe Steemit is going to blow up!

@steemitdev Great work! I do have 1 question though. I am a witness in position #60 and you said;

" We discussed this decision with the witnesses and they were supportive of this change. We are still planning on implementing and releasing HF 20 in the coming months."

But I don't recall ever discussing this with anyone on the dev team ? Can you please elaborate? Thanks again.

·

Inquiring minds also want to know. Thanks for asking this question!

This is a great start. Thank you for acknowledging the scaling issue. Thank you for working hard to deal with it. Please devise a strategy for maintaining and expanding on the information you present to the community. All the best.

These kind of updates make me feel very confident in the platform and it would be great to read development news and plans on a regular basis. Thank you all for your great work!

Brilliant, just brilliant! Now can you direct your attention to a very pressing matter: Please program a CANCEL button for the ones that want to leave Steemit. If you can produce a REGISTER button, then you can also program a CANCEL button. As another user here stated: It is not possible to CANCEL on Steemit, because everything is written in stone on the blockchain. I agree. But as the old Romans already knew, EVERYTHING can be chiseled into stone, even a CANCEL procedure.

And while you are at it, maybe you can have a few thoughts in your young brains what happens if a member of Steemit dies or wants to sell his/her Steemit account. After all, this is an asset like every other asset and therefore I must be able to gift it, trade it or pass it on to my kids.

Thank you.

·

Well, you can gift your Steem, if you have time to power down.

I actually, to my surprise, agree that heirs, assigns, and survivors is an issue regarding power down and probate, at least.

·
·

Well, then I would appreciate it if somebody would attend to those issues. They will come up in future anyway and then there'll be big tears and legal cases.

Nice info, is there any place for proposals? New users really get bounced off by the fact they have to put an url for their avatar image, to complicated. Greetingz from fellow developper

It's good to hear that work is progressing. There will be challenges is scaling up Steemit. When there are millions of users it will get really interesting. At least you can learn from what others have gone through.

Please keep us updated

Finally the change we expected :)
This is good news :) thanks

I'm no expert but mightn't it also help if there was a no-javascript ultrabasic version of Steemit running as well that people on mobile and/or low-bandwidth connections can use?

Might lighten the load on both ends, is what I'm saying.

Its very good news)))

Thanks very much for this nice social network.
I mean its realy cool for connection people idea....

Congratulations @steemitdev! You have completed some achievement on Steemit and have been rewarded with new badge(s) :

Award for the number of upvotes received

Click on any badge to view your own Board of Honor on SteemitBoard.
For more information about SteemitBoard, click here

If you no longer want to receive notifications, reply to this comment with the word STOP

By upvoting this notification, you can help all Steemit users. Learn how here!

Great descriptions of what you guys are working on now. Enhancing steem/steemit's capacity for handing traffic seems like a great focus for this moment as when the flood gates open we all want the product to work seamlessly for all of the users new and old. Beyond this it will be absolutely necessary in the Advent that steem/steemit is eventually made into a WordPress like plugin that can be implemented everywhere someone needs commenting/posting abilities. Thank you for the update.

this is great news and puts to bed some of the worrying rumours that were circulating regarding scaling.
Time to buy some Steem then :)

Easy to understand post and great news too! Steemit just continues to improve!

Trying to buy STEEM this morning but failing. Any ideas?

·

All seems good

Great work steemitdev. I like the "Scaling the blockchain". Using one stove with 8 burners is much easier and faster process to making progress than the delay on one burner. I see these having a great positive impact on steemit. Good job, great team, awesome result.
Good luck guys.

I can't wait the see the future of Steemit a few years down the road.

Nice post broe.

Thanks for the diagram ..my husband will also be interested in seeing it !

Nice post

Thanks for this. Very helpful indeed.

Thanks for trying for all users, We hope goods in future

Great stuff. I hope this part:

"in favor of enhancing our application framework."

means you'll be addressing the API complaints I've seen. Thanks!

Hi very good . You also follow me please

Great update. Thanks!

great work! gonna be flying soon it seems! :)

So i guess, we need to maximize our burners to become more efficient?

(1% knowledge in blockchain. xD)

Wonderful post! This is the first time I am seeing the graphic you posted as I am newbie to block chain and cryptocurrency so thank you! #followed

I don't understand it,but am glad to be a part of it!

great work i am slowly learning coding on codecademy to better understand

muy buen contenido me gusto tu post te felicito, éxitos!!!!

Great pos, thanks for sharing, regards

woow,,, amazing post friend... I like this @steemitdev

The key problem is that Steemd is single threaded, however, we are well along in the process of making it multithreaded.

This is fantastic. Looks like Steem will be the first multi-threaded blockchain ahead of EOS.

I did not see I/O mentioned - this seems like a greater challenge currently. I assume you're working to optimize it, but it would be great if you mentioned specifics.

·

This is fantastic. Looks like Steem will be the first multi-threaded blockchain ahead of EOS.

Naturally. Our team who actually conceived and designed multi threaded scaling with Chainbase and the rest of the steem blockchain stack have always and still work here. I'd be surprised if any group copying the strategy got there first.

·
·

Multithreaded? could you kindly explain more on this? is this the same concept in Computer Science e.g. multithreaded Programming?
Thanks, @justyy

·
·

I wasn't sure I had read this part of the post right. I'm glad you're confirming this part.

really is great news

Followed all in this thread upvoted and Resteemed

Great news!

Sounds Fantastic.

Scaling being addressed before it snowballs out of hand. "SMART".

Great so, When is HF20?

Hi guys,
there are a lot of users has an issue - it appear today.
we cannot attend our blogs with direct links , say steemit.com/@allfabeta.
The steemit answers with a robot picture and message "The blog has been removed" or link is broken

I can only access my blog from a main page, then lookup for any my recent post by tag or with lookup and then hit on user's name

https://steemit.com/steemitbug/@allfabeta/steemit-issue

I'm connecting from xfinity USA provider, others how have the same issue from Russia

Great advice very easy to follow.

easy to understand with your format

Thanks for sharing Steem blockchain updates.

Steem Blockchain update is awesome
The new format its easy to understand.
Thanks to @steemitadventure and @overkillcoin

Great that you are addressing the scaling problem. Any specific design pattern in play here?

I LOVE THIS ,,,,, steemit rocks ,,,

That's good news, good post, hope bettet for future, you do so good @steemitdev

Keep Up The Good Work Guys :)

Excellent, very glad to read about these positive developements towards multi threading and moving tasks to plugins.

Keep up the good work, ReSteemed!

how will you make sure that Steemit stays decentralized? I ask because you mention the reliance on Amazon AWS.

Cool, so from the sound of it, this should make the Steemit site faster, right?

Hello, nice posting, i like it and i have restem your write ..... i'm @mukhtar.juned from Aceh Indonesia, please give me the tricks for me to get a lotv upvote from steemiters world, I can not write because I am just an employee who daily just work on schedule from the company. Give me your advice so I can go forward like you, thank you.

Best regards
@mukhtar.juned

Thanks for the update! Sound like you have a fun job!
I'm amazed by the fact that Steemd was single threaded. Nice to hear that you are refactoring.

I was wondering when that silly single-threaded daemon would go multi-threaded. I'm glad you guys are addressing that, finally! Replays will greatly benefit from it.

That's Great!
Hi Guys, today 1 Bitcoin = $3,448 Have you been interested investing in crypto currency,
but you don`t know where to start...
Learn how to trade Bitcoin and all Cryptocurrency! and Earn Passive Income!
Secure Your FREE Position NOW! Click Here to Get Instant Access
http://bit.ly/2v3Stem

Guys, please upvote I will do the same in return.

Please Follow me : @bogz
Please Follow me on Tweeter/Instagram: @bogzky
Add me on Facebook : https://www.facebook.com/bogz.catangay
Please sign up : http://bit.ly/2v3Stem
Like my Page : https://www.facebook.com/FutureNetPH

Team Steemit, keep up the good work. Scaling and multithreaded bring back my IBM days for large coporate data center applications. Looking forward to your updates and please with my experience on steem so far. Upvote resteemed and following

the analogy was brilliant! It certainly sounds very complex, as it appears you are all having to create a system as you go....Thanks for the great update.

thx, for the update, keep up the good work.. after 2 months I love the platform even more..

Where is the best place to mine STEEM?

your post is very good. It is easier to understand.

Thank you for taking the time to update the community. I think it is important to share with us the developments, even if it is somewhat technical, especially when there has been a bit of negative sentiment lately around the prospects of the site!

Really well written and I believe you made it fairly easy for those that aren't to keen on coding as well as what goes on within the backend of Steemit to be able to understand.

Thank you for the update!

can u give some examples of a plugin one could run?

Hi @steemitdev! You have just received a 0.1 SBD tip from @cardboard!

@tipU - send tips by writing tip! in the comment and get share in service profit :)
By upvoting this comment you support the service - thanks!

Good!😃😃😃