Proposal: Public Hivemind SQL service

in proposal •  4 months ago  (edited)

Hivemind, a tool by Steemit inc., is a backend API service currently replace some of the endpoints of steemd. It's in use for a good amount time.

I've been running a public Hivemind node, distributing daily database snapshots almost 9 months. There is also tower, a REST interface for Hivemind data, is developed and maintained by me, so far.

RPC and REST interfaces are just the tip of the iceberg since the underlying database -PostgreSQL- offers much more flexible queries. (Check some examples)

I want to offer read-only public accounts to query the Hivemind database. This requires new infrastructure planning. Can't offer public accounts on the master PostgreSQL instance due to obvious reasons, I need new at least one or two replica instances where people can query and benefit the power of the Hivemind.

These come with additional development/maintenance/server costs. While I am happy to do that with my witness earnings, I already run a lot of dedicated servers.

Didn't really do the math, but probably get into a loss if we compare the costs with witness earnings. Here is my server list for STEEM related things:

  • Two dedicated servers for witness
  • One dedicated server for Hivemind
  • One medium-sized virtual private server dPoll
  • One small-sized virtual private server for scripts (witness backup failover, etc.)

TL;DR What will be delivered?

A reliable, fast, public SQL service (direct connection) of the Hivemind. I expect to deliver it before 2020. The plan is exposing Hivemind's database directly to the public with a good, old PostgreSQL user. (read-only)

Timeframe of the funding

1 year

Daily request amount of SBD

12 SBD


  • What will happen when the funding window is closed? Will you shut down the service?

1 year is a lot, I cannot promise anything at this time. But, worth to mention, I already served free public Hivemind node for 9 months, and I expect to keep this service is up as long as I can, financially.

  • Will you cash out the funding instantly?

My plan is to power up the SBD, instantly after converting to STEEM. I will cover the server costs with fiat and don't create sell pressure on STEEM with the steem dao funds. However, this might change in upcoming months depending on the costs, my financial status, etc.

  • Are you eligible for that kind of thing?

I am a software engineer with 10+ years of experience. Also, experienced with Python (the language powers Hivemind), and PostgreSQL. (My github account @emre might be a good indicator about what can I do.)

  • What will be the outcome for community?

This is useful mostly for software developers, data people. Applications, dapps, reports can benefit this direct SQL access.

  • Difference between SteemSQL?

SteemSQL is a general blockchain data service, while Hivemind only stores a partition of the blockchain data. Mostly social part like posts, accounts, votes, relationships (follow, ignore, resteem), etc. It's also planned to store communitiesdata at Hivemind.

Voting for the proposal

You can view all the proposals on:

If you have any questions or suggestions, shoot in the comments section.

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

I'm in ;)

@proxy.token supports your proposal !! I've already upvoted.

Thank you, that's great news!

Güzel bi aksiyon Üstad, eline sağlık. Bütün oylar senin olsun.
Keşke yazılımcılardan parti kuran çıksa da siyasi oylarımızı da ona versek ;)

Tesekkurler Murat :-)

Selamun Aleyküm Katılıyorum sisteme yeni katildim ancak boyle basarili turk kardeslerimi az goruyorum birbirimize destek ciksak daha iyi olur diye düşünüyorum. Başarilarinin devamini dilerim.

@organduo A good one to vote on. Will be helpful for our @untersatz witness activities.



@emrebeyler you have received 5 SHADE from contrabourdon!
View and trade the tokens on Steem Engine.

This tip bot is powered by witness untersatz!

voted :)


Please explain to the average guy like me who has no idea of what your saying

Outcome of this project is that software developers, app owners, data people can have access to blockchain data via something called SQL.

SQL helps to query the data in a practical way in comparison to what we have in public resources.

So you wanna implement it on Steemit?

Not sure if I understand the question :)

Just curious how many people are using your hivemind? or how many services?

Didn’t check the numbers so far since it has a health check monitored and running flawlessly since its lunch.

Regarding apps, and uses Tower. Also my recent tool uses my hivemind db.

This would be awesome.

Thanks for the support 🎉

20.05 USD has been spent to promote this content using Steemium .
Learn more here!

Is this similar to the @arcange database but for PostgreSQL?

Posted using Partiko iOS

Yes, it will be the same except Hivemind doesn't store whole blockchain data like SteemSQL. Instead, it focuses on social aspects like votes, comments, posts, resteems, relationships. (ignore, follow, etc.)

In the future, communities data is also planned to store at Hivemind.

Sounds great! Looking forward to be one of the first to test :) You already have my witness vote and I will support this proposal.

Thanks! 🎉

Isn't that data available through steemSQL too?

It is available with a subscription fee.

Is that the only difference between what you propose and steemSQL?

Post includes a Q/A section explains the main difference.

SteemSQL is a general blockchain data service, while Hivemind only stores a partition of the blockchain data. Mostly social part like posts, accounts, votes, relationships (follow, ignore, resteem), etc. It's also planned to store communitiesdata at Hivemind.

So your service will only offer a small part of what steemSQL already offers but is funded by the community? I am pretty sure that you can get follows, ignores and resteems from steemSQL as well. I fail to see a true benefit tbh.

I run my own hivemind node as well as pay for SteemSQL access. I find that hivemind is often better for querying certain things as compared to SteemSQL.

In some cases, results come back faster in hivemind because of how the indices are arranged.

One of the major benefits of hivemind is how tags are tracked. If you’re doing content specific queries, it’s usually better to use hivemind.

I have used SteemSQL, it is awesome. However, it is not affordable for an average user, especially with the current Steem prices.

Building a free alternative for users and independent developers should be welcomed.

  ·  4 months ago (edited)

Where am I twisting words.

  • Does steemSQL offers everything you want to build? If not what will you provide that steemSQL does not.
  • Do you want the community to fund your project with the SPS?
  • Is steemSQL paid only by those who uses it?

I don't see where there are any twisted words. It seems you just can not justify the usage of the SPS to build a service that will benefit the 1% on Steem even though such service does already exist.

All this seems to be is a cheap steemSQL clone funded by the SPS. Nothing more. Nothing less. The only "toxic" answer here just came from you because it seems you have no good arguments left ;)

  ·  4 months ago (edited)Reveal Comment
  ·  4 months ago Reveal Comment

Thanks for the reminder about tower.

I'd like to see a hive mind all db.

Posted using Partiko Android

Then you should definitely vote for the proposal! :-)

Yes I should, you are right 😁

Posted using Partiko Android

I have voted for your proposal.

What kind of hardware resources would I need if I wanted to run my own copy of Hivemind/Tower?

Thanks for all the hard work you do.

I would suggest a dedicated server with at least 16gb ram and 256gb SSD. For me, I run it at a box with 32 gb ram and nvme.

Your proposal is lacking... to 99% of us.
It seems to be a proposal written directly for developers... and maybe that's smart because maybe you're only here to get the votes of whales and developers but you could at least tell the rest of us why we'd care about SQL hivemind and how it will impact our day to day activities... or don't because our votes aren't gonna really get you anywhere and maybe you only care about the votes of those who know the value of this database you're talking about.

Anyway thought you may want to know what most of us think about these proposals. We want to go through and vote on stuff but then we see that the proposers just don't care to write a post directed at us. But then again they're all being done by developers and I mean that's not their strong suite i guess. So i mean I'm not castigating you (you're doing what you need to do) i'm just saying it just sucks for us.

That’s a good feedback, I’ll try to explain these stuff better if I create another proposal in the future.

You have only me to blame for bringing this post to the trending.
Sorry if the post didn't include an explanation for non-dev people.

To sum it up, This lower the barrier of entry for developers to start experimenting. I don't see it as subsidizing resources but more as a playground to later develop more serious stuff that won't depend on this service.

You got a 78.95% upvote from @minnowvotes courtesy of @steemium!

Congratulations @emrebeyler!
Your post was mentioned in the Steem Hit Parade in the following category:

  • Pending payout - Ranked 2 with $ 98,21

Voted, what a great idea


Thanks for the post.

I voted your proposal!

@tipu curate

Upvoted 👌 (Mana: 0/30)

Support this!

Posted using Partiko iOS

Thanks! 🎉