Hivemind is Live!

in #hivemind4 years ago

Hivemind Live.jpg

Hello Steemians, we are excited to announce that Steemit’s APIs are now powered by Hivemind! By routing most of the social APIs to Hivemind, we are able to dramatically shrink the size of the full nodes we run. Because there are so many social applications sharing information on Steem, full nodes had become extremely large and expensive to run especially because they are not optimized for use cases that are not consensus-critical.

Hivemind is a service that syncs a traditional database with the blockchain. An added benefit of Hivemind is that since it’s written in Python, developers can modify its behavior and APIs with ease, instead of relying on C++ developers and modifying code which can interact with critical consensus logic. Implementing features like unreblogging becomes trivial with Hivemind while simultaneously enabling node operators to dramatically shrink the size (and cost) of their servers.


In our tests we were able to route all APIs to the new configuration—Hivemind for social APIs and everything else to “slimmed down” steemd nodes—which performed well. In addition, these nodes were running on-disk, as opposed to RAM; a major win which was only made possible thanks to AppBase.

We are extremely excited to see Hivemind running in production, on-disk, and utilizing AppBase. After completing this task, we have decreased our steemd instance sizes from 488GB RAM to 61GB RAM instances. We are doing further analysis and may be able to decrease this usage even further. These are major improvements to Steem that will make it easier and more cost efficient for anyone to run a full node, and all without requiring a hardfork.

Bug Hunting

We have so far identified one minor stability issue and will be rolling out an update shortly. If you are experiencing any issues using please leave a note in the comments.

The Steemit Team


Awesome. Great job delivering this guys! I was very glad to see this route taken as opposed to just 'sun-setting' the most popular and widely used app in all of crypto :)

This is fantastic work! I wondered whether you’d really be able to get RAM requirements down below 64Gb, but you’ve done it.
Next question: who wants to help me set up a full node. I have the hardware to meet the 2x64Gb + 1 x 32Gb requirements and have very fast internet 300/150 Mb/s.

Posted using Partiko iOS

please contact me, @surfyogi on telegram messenger

A full node with only 61gb of ram is the dream ! Congrats team ! Can you guys share more specs about your instances ? How much disk, ssd or nvme, which cpu etc ?

 4 years ago (edited)

We run two types of Steem nodes - one that handles account history, and one that handles everything else except for the tags and follows plugins (because Hivemind handles these plugins itself now). Both types of nodes run on instances in AWS that have 61GB RAM and a single physically attached nVME drive. The shared memory files are stored only on the physically attached drive, no longer in RAM as was previously required. Hivemind itself uses a postgres database in AWS RDS on instances with 32GB RAM. Hivemind's 'app servers' use much smaller instances with only 4GB RAM. All of this has been a phase in our plans to streamline infrastructure for cost effectiveness and there may be further improvements still yet to be made.

So, how many moving parts has Hivemid API instance?

  • app server
  • postgres
  • api node


 4 years ago (edited)

Yes, hivemind requires a postgres database. You need a steemd node or nodes that contain all plugins except for tags/follows. We use jussi for routing, which is a custom reverse proxy and caching layer.

Everything has official docker images available and could be set up to run together with docker-compose.

Well, I’m not a developer guy, I just wanna figure out how much you save on old rpc node vs new set of nodes delta.

With the cost of instances with lots of RAM being very high and fast disk being relatively cheap, quite a lot.

How much 1Gb RAM costs now?

5 comments in the first 10 minutes... This place is lively... 🕸️🍃

I ran into the gateway error when trying to load steemit for some minutes after the post went up. Could be why.

Nice to see you @beanz! :D

Howdy Stranger.

Don't worry, it's been very lively around here in the past few weeks.

Posted using Partiko Android

It only looks lively for people who haven't been around lately :-)

Shhh everyone! Beanz is back.

I hope you are doing well and all things are well with you. To quote @steevc:

We've missed you...

That is a very positive news to say the least!
We left a note earlier today on a prior update by @steemitdev, whereby we believe one related functionality in nodes is broken, as follows:

hey there,
We noticed at the time of rollout of changes (as of Jan 10) that the post's rshares data is now returning undefined
Is this an intended behaviour?
We are for example pulling post data using steem.api.getDiscussionsByCreatedand then accessing post's post.vote_rshares
This is only no longer working on and
Would be great if you can provide some insights or look into this.

Yes I am also very pleased and want to see communities man, i just want subreddits , just something simple like chainbb, when it finally happens and we have an instant reliable faucet for creating new accounts then steem can finally just start growing organically on its own JUST from all the reddit users who will Migrate over just for the censorship resistance AND the payouts! In fact I think the censorship question will be enough to bring Millions of users over since that is what happened when Digg users first migrated to Reddit and then reddit users migrated to Voat and etc

We are some of the first wave of reddit to steem immigrants and we are like the paratroopers who were dropped in over enemy lines into Normandy to capture strategic bridges etc and soon we will have a Normandy type invasion from the Bitcoin and crypto subreddits first into steem and then all sorts of other subreddits like r/trees r/TheDonald and all sorts of political subreddits, photography subreddits looking to get some money, meme crafters who post normall on reddit and 4chan for nopthing will find Dmania and take it over and also build their own versions to monetize memetics

its going to be a bright future for steem, we have riden out the bear market and its actually made us much stronger

my god I hope we don't get attack by the redit crowd....
they are nasty at times....

Are you referring to rshares per vote, or per post? If the latter, the net_rshares field is the canonical source.

Yes the per post one.
So are you saying the value of vote_rshares has been dropped, while the net_rshares still has value? I am unable to test this out on or as I keep getting an RPC error atm

Try again, there was a small hiccup ;)

works now, and yes net_rshares is returning value, thank you.


This doesn't sound right. :D

This is a great update, though. Thanks for all the work on Hivemind, that sound like some major size decrease!


de/unsteeming :P

I think that upvote ===> un-vote.
Resteem ==> un-resteem.
So it is un-resteeming.

I like unsteeming

I could wrap my head around deresteeming. I think.

Is that what Secretary Clinton did? I gotta learn how to do that kind of high tech stuff.

Great work! Hope this will make it feasible for dApps, or collaborations of some projects, to run their own full nodes in the near future:). Certainly a top priority from my point of view. Well delivered!

If you are experiencing any issues using please leave a note in the comments.

Encountered the typical 504 Gateway Error for maybe 5-10 min prior to reading this post (so roughly 40 minutes after it was published) if that is of any help. No other issues experienced yet.

Good that means it is working yeah how can I sleep now🙈 I'm all fired up on this. 🤣😂

Posted using Partiko Android

Actual footage of Steemit HQ and Hivemind coming online

@mstafford, you and your Python skills will be taking over the world..

Python for the win! Gots ta keep it simple!

Sure. Even we economists know some python. Not that weird cpp stuff.

Python is huge for financial scripting!

This is a nice development. Good job steem development team.

For the past 10 minutes I have been getting a lot of 501 errors for steemit pages, no problems surfing elsewhere on the net, just steemit. 12:45 pm-12:55 pm Alaska standard time.

These issues should now all be resolved.

hi.... i'm not a programmer and i dont know about hivemind, can someone explain to me how it's affected/ benefit to steemit in understandable language ??

Steem (busy and Partiko) was sometimes quite uncommunicative over the last few days.
Sometimes not loading at all busy & Partiko.
Sometimes not accepting votes or comments (request too long/ time out)

The Wallet Transfer list is also pretty short.. But that may be intended.

Posted using Partiko Android

I believe busy does not rely on our infrastructure. I am not sure about @partiko

Yeah, that explains a lot. Thanks for the update!

Also, I know a lot of people are concerned about the silence around the power down from Steemit Inc, yourself, and Dan. Would you be willing to comment on it? Because honestly, any communication is better than none. Hell, it'd help with all the FUD getting thrown around last night...

That’s what I’m talkin’ about! Progressing! Now let’s 10x baby!!! 💪🏼🙌🏼💯

Posted using Partiko iOS

10X, you must have just read a Grant Cardone Book XD

Grant gave me one personally 😂

Posted using Partiko iOS

Great to hear this. Does this mean that hivemind functionality will be available through steem public nodes and will soon be implemented in the documentation ?

OK,,,does this do anything for increasing value of $STEEM? Or bring a larger adoption by the un-tech among the rest of us?

Not so much!
The people that can comprehend what is being said here, are largely already invested in Steem ;-)

Woohoo! Congrats. Sounds like we just got upgraded

I Love it even though I could not understand what it all means I can still the bright spirit emanating from this news @steemitblog
God bless your works.

Whoooooowwww @exyle check this out... I'm gonna rent me another vps to run a full node 🙈 I have to work tomorrow o no.. great... I order it tomorrow. I just reacted on your post 🤣😂🤣😂 yeah! Good job!

Posted using Partiko Android

Even though I don't understand a thing, I'm happy for the updates. Keep them coming!

Hurrah! That's great news. I thought there was the odd error here and there and figured something was going on under the hood.

It's actually good knowing that something is going on under the hood ;O)

Hivemind is ready that is huge news!
Does that also mean that the long awaited community feature will be available soon?
Great update guys!!!

Posted using Partiko Android

The communites feature is a priority in the new Mission, Vision, and Values that can be seen on the about page. Also see Ned's post from earlier today.

NO REAL INFO on the About page bro ;-)
But thanks for all your comments, it's much clearer to me now...
I am missing the "How to become a witness" link, and the page the specs the future job of witnesses, now that it does not require a lot of money to run a node, I wonder if that might remap the witness roster, as far as WHO are really willing to be the most professional witnesses?
AND what does that really mean; should witnesses be responsible for promoting the network as well?

Pieces are falling together!
Yep, there is this bug when you check a wallet - you can't go back to your blog page/ or any other by clicking buttons (reload is the only way).


I'm a victim of this to.

Thanks for the report, fix incoming!

Great Scott! Been waiting to use that one for a while! Lol 😂 really hope this does bring some improvements coz hot damn the moral in this place is lower than the current price

Posted using Partiko iOS

This is actually wonderful news. Great work guys.

Probably the best news we had from you guys...for a while

Holy cow, that is quite the reduction! Thanks for sharing this with us.

I have been encountering annoying problems during the use of Steemit.

First of all, I do not see all the posts browsing the given tag, even in the feed tab I do not see them! I checked on busy and steempeak - there were no problems.

Secondly, by going for example on the wallet tab and going down to see the transfers, instead of them I see the first part of the wallet (balance, savings, etc.) is shown to me 100 times and then the history of transfers. The same occurs when I click on the curator or author's rewards. I see my wallet 100 times, then rewards. And this is not a fault of the device, because I checked the steemit on my main computer, laptop, tablet and two phones - everywhere the page is bugging. For now, these are the only mistakes I have noticed but very annoying. I hope that you will be able to fix it in the near future.

Thank you @steemitblog :)

Posted using Partiko Android

I haven't been able to reproduce this on your profile. Does it persist?

I even tried to delete cache and change internet browser. Same :(

Posted using Partiko Android

Identified the bug, fix to be deployed shortly. Thanks!

No problem Sir! Have a great day :)

Posted using Partiko Android

Haha. Yeah. Exactly the same thing. Multiple copies of the wallet when scrolling down.

I am confused: Wasn't Hivemind supposed to be the long-awaited community-account feature?

Hivemind is the software that can make the communities feature possible. It is now running and serving content for, which is a major accomplishment. This means that the communities feature could be built and implemented - along with other types of social features that were previously non-trivial to implement at a consensus blockchain level. This opens the doors for much easier and faster development of features that every day users can actually see.

As an added bonus, it is also more cost effective to run than to soley rely on the blockchain for non-consensus data.

Does that mean we will be getting communities feature sometime in near future?

The communites feature is a priority in the new Mission, Vision, and Values that can be seen on the about page. Also see Ned's post from earlier today.

Ok, thanks for explanation.

I'm starting to get somewhere with my graphql api that uses hivemind as a backend. If anyone wants to check it out I've kept the developer playground open at Let me know if anything breaks. So far you can query posts and accounts

While this is great news and essential, when is Steemit Inc going to communicate with the community about the massive powering down from Steemit owned accounts as well as the personal account of @ned ?

This blog post makes me feel a little better that STINC isn't making a bank run and abandoning the community, but still, without an answer, I can't help but feel the movement of seriously large amounts of STEEM including powering down $8.5m USD worth on the @steemit account.

What is going on?

Hey now, you know what they say... @beggars can't be choosers. You got hivemind, that's something for now. ;)

Hey now, you know what they say... @beggars can't be choosers. You got hivemind, that's something for now. ;)

Hey now, you know what they say... @beggars can't be choosers. You got hivemind, that's something for now. ;)

A hobo and a beggar walk into a bar. 😂😂😂

Nice to see progress. Resteemed :-)

Thanks for an amazing progress. I hope this will get us further in the long run and will help to compete with EOS in terms of feasibility of running your full node.

Long awaiting news of any kind. They seem very positive indeed!

Will go next and see what Ned said in aggroed and llfarms show. Hope he addressed the power down thing more at large.


It works well, the transition was smooth. Congrats

Thanks! A lot of hard work on the part of @roadscape and the rest of the team.

I thought you migrated to LINO. 😂😂😂 Nice to see youre still around.

What is LINO?


DLIVE network

Although I do not respect some things they have done in the past, they may be much more successful and thus should be considered. Left-coast mentality has traditionally been much less "encumbered" by East Coast moribund financial constructs that limit the "out of box" thinking in my view.
Thus the historical lesson and metaphorically; Microsoft was a horrible example of high tech community building in so many ways, but it was a great investment (at one time), and incredibly successful.

Such a ram optimization is very impressive!

Impressive move by the team... Congrats everyone on the platform.

Posted Using - Supersteemian Android Application

Great progress.... Now I need to start learning stuff to understand what this means... lol

regards @paulnulty
When you read the first 20 or more comments, you will learn a lot, a master class of valuable information. We are already two hahaha.
(I use a translator)
Al leer los primeros 20 o más comentarios, aprenderá mucho, toda una clase magistral de información valiosa. Ya somos dos jajaja.
(Utilizo un traductor)

Gracias por la informaciòn. Publicaciòn de mucha utilidad digna de ponerse en pràctica.

Could you do a follow up post related more to the meaning of/implication of this down the road in both the immediate and long-term future of the tool? Thank you thank you!

Great news. I will a time to building a server for my hivemind.

keep up the good work....

Bravo!!! If you do not understand much or nothing, but you love the heat (enthusiasm) generated by this site ... Vote for everyone and subtract the post. And to study.
Bravo!!! Si no entiendes mucho o nada, pero amas el calor (entusiasmo) que genera este sitio... Voto por todos y resteemeo el post. Y, a estudiar.

Coin Marketplace

STEEM 0.18
TRX 0.05
JST 0.023
BTC 16939.43
ETH 1274.62
USDT 1.00
SBD 2.09