SteemSQL Wrapper — analyze any Steemit data right in your browser with one click

in #utopian-io7 years ago (edited)

You see all these analytical posts, want to start creating something similar, but have no idea where to start? Now you can!

Problem

Steemit has really a great infrastructure — many various developers are working on great projects, some of them extending features of Steemit (like steemstats.com or Vessel by @jesta), some creating something absolutely new (like utopian.io by @elear), some helping users and giving answers to their questions (like steemwhales.com by @heimindanger).

And it also has many great projects for developers (e.g. SteemSQL by @arcange, Piston by @xeroc, or SteemJS by @fabien). The problem is that all these projects are not tolerant of non-tech people. For example, what if you just want to get info about the most upvoted posts in the past month? While I was writing my first Steemit post, I looked at all available tools and now can allocate several options:

Writing a crawler

It is very inefficient. And what if you want to get info about the most upvoted posts in January 2017? I just want to note that in this case, you will need to crawl more than 3.5M posts (more precisely, 3,561,711).

Piston/SteemJS/steem-python

It is great if you want to do a complex analysis of absolutely all Steemit data. But again, it is very inefficient in other cases — you will need to get data block-by-block, meaning that in order to create a list of the most upvoted posts in January 2017, you will need to get all blocks from today.

SteemData

MySQL right now is not available, API is not very flexible (but I think that in several cases it will be the best solution). MongoDB is NoSQL, meaning that Mongo has its own system of queries and does not support SQL. I personally do not have any experience with Mongo or any other NoSQL databases, so in my case, this is not the best solution. I believe that for the most non-tech users it is also not the best solution because SQL has many various guides and tutorials (I guess it is about dozens of thousands) while Mongo has significantly less popularity (this is logical because SQL is the standard and it is used by many databases). But if you have any experience with Mongo, then it will be a great solution.

SteemSQL

I personally believe that this is the best solution for the most analysis. At least in theory. In practice, it also can raise some problems. For example, it requires drivers for SQL Server, but installing these drivers from Linux or Mac is a real adventure; Microsoft even does not have any notes that they have these drivers for Ubuntu 14.04 and 17.04 (moreover, all answers on askubuntu, serverfault, stackoverflow, and on any other similar projects say that Microsoft has drivers only for Ubuntu 15/16).

But the biggest problem for non-tech users is that SteemSQL has only two options for use: programming languages and Excel. You cannot just read a quick tutorial about SQL, start experimenting while you are learning, and get the first results in less than an hour. Also, Steemit has many mobile-only users who cannot use any of these tools even in theory.

Solution

So I decided to create a solution — SteemSQL Wrapper

out.optimized.gif

It allows use SteemSQL right in your browser without any dependencies. It even allows using the several most popular queries with one click! It means that non-tech users no longer need to install any drivers/Excel and learn programming languages. They can just go to SteemSQL Wrapper and write any SQL queries to get results right in their browser. It also has the link to the SQL Tutorial for users who do not know SQL.

out2.gif

But it is not everything. The important part of the project is the possibility to get results even without knowledge of SQL — I prepared the several most popular queries and anyone can use them with one click. These queries include:

  • Тоp 30 posts by number of upvotes
  • Top 30 posts by number of comments
  • Top 30 posts by pending payout
  • Daily number of posts
  • Daily number of new accounts
  • Daily number of votes
  • Top 50 utopian contributors by pending payout

I will extend this list in the near future and if you want to add any specific query then just write me.

out3.gif

Future features

In fact, they depend on your feedback. But right now I have plans for the next features:

  • Easy sharing of queries between users
  • API that will allow developers to send a simple GET request with their SQL and get JSON with all data
  • A simple interface which will show all tables/columns from the database
  • Autocomplete
  • History of your queries
  • Analysis of received data (sort and calculate averages/medians/sums of columns)
  • Ability to create custom graphs from data

For Moderators

My commits:

image.png



Posted on Utopian.io - Rewarding Open Source Contributors

Sort:  

Very cool project!, this will be very useful and interesting to follow!

I would love to see the future features!

Thank you for your submission.
Your contribution cannot be approved yet, because it is in the wrong category. See the Utopian Rules.
This fits better under development work, provided this is your original work. To be accepted, you need to provide proof of work, but also link to your related github repository along with relevant recent commit links.

You can contact us on Discord.
[utopian-moderator]

No problem, but I have a few questions.

  1. Any post of absolutely every subproject will be related to the development. So, what is the difference between Development and Sub-Project categories? Could you describe what exactly you want to see in the Sub-Project category? For example, if I will develop something like DTube, ChainBB, or dMania, what will category be right?
  2. I thought that the Sub-Project category is for launching projects while Development is for further development. For example, if I launch a new project then I write a post in Sub-Project about reasons to create it and detailed description (about an audience, roadmap, main features, how to use, etc). But when I add new features or fix bugs then I need write about it in Development. Is it not so?
  3. If I will follow your logic then all the current reviewed projects in the Sub-Project should be in the Development category. They all are related to the development. Am I not right?
  4. Are my username and the link to my Steemit profile on the homepage of the project not enough to proof of work?

Hey no worries. Thanks for inquiring
sub-projects describe, in great details, major functionalities "suggested" to be implemented as part of an existing project.
If you check out Utopian Rules you will see the top rule as
A Sub-Project is a set of new unseen features having a great impact on the project.
keywords are unseen, and great impact.
If you check out the remaining rules, as I mentioned earlier, the features need to be very well described, with screenshots, illustrations where possible,...
Sub-projects are suggestions for projects that you do not own, it doesn't make sense to come up with a proposal to yourself :)
Development category on the other hand is for work that has been already developed by you, with proof of work and links to github, PR, and commits.
Hope this clarifies it well, and feel free to reach out if you have further questions

You can contact us on Discord.
[utopian-moderator]

Ok, got it. And did update the post.
The last question: the difference between Suggestion and Sub-Project is only in a size of suggesting features? Minor features are in Suggestion, big features are in Sub-Project. Right?

Yes that is correct.
We actually still need one item from you. Since your utopian name is different than the github one, we need proof of work, confirming that you are the author of the github.
You might want to edit the readme file and include this detail, and attach a screenshot confirming this here.

Thanks for the explanation.
About the confirmation — just to clarify, rules say:

If your username on Github does not correspond to the Utopian username you must provide proof you are the owner by providing a screenshot of the logged in session in Github.

And I provided this screenshot. Also, an author of this post provided exactly what you want but his post was rejected.

Why are requirements of all moderators so difference? Anyway, I did update the repo description.
image.png

Thanks, accepted now.
Mods have general rules to follow. Sometimes they make mistakes, we are all human. In any such case, feel free to reach out.

Perfect post.. Thank you very much for the very useful contribution

This post has received a 11.34 % upvote from @booster thanks to: @emptyname.

Congratulations! This post has been upvoted from the communal account, @minnowsupport, by emptyname from the Minnow Support Project. It's a witness project run by aggroed, ausbitbank, teamsteem, theprophet0, someguy123, neoxian, followbtcnews/crimsonclad, and netuoso. The goal is to help Steemit grow by supporting Minnows and creating a social network. Please find us in the Peace, Abundance, and Liberty Network (PALnet) Discord Channel. It's a completely public and open space to all members of the Steemit community who voluntarily choose to be there.

If you would like to delegate to the Minnow Support Project you can do so by clicking on the following links: 50SP, 100SP, 250SP, 500SP, 1000SP, 5000SP. Be sure to leave at least 50SP undelegated on your account.

This post has received a 0.18 % upvote from @drotto thanks to: @banjo.

Congratulations @emptyname! You have completed some achievement on Steemit and have been rewarded with new badge(s) :

Award for the number of upvotes received

Click on any badge to view your own Board of Honor on SteemitBoard.
For more information about SteemitBoard, click here

If you no longer want to receive notifications, reply to this comment with the word STOP

By upvoting this notification, you can help all Steemit users. Learn how here!

Hello @emptyname, this is really something useful :-D

Could I suggest 2 improvements :

  1. Use (NOLOCK) in your demo queries, this is something @arcange really insist on to new users of SteemSQL. It also improves slightly the performances (ask @arcange for more details about it).
  2. Could you also provide some easy to use information about the database structure (list of Tables/View + available fields).

Anyway, great job and thanks for sharing.
This tool is easy to use, I really like it.

Hey @oroger, thanks for the feedback and suggestions!

  1. I know about NOLOCK and did read posts of @arcange about it. But now it is optional
  2. Yeah, it is in plans (check the Future features section of the post).

Hey @emptyname I am @utopian-io. I have just upvoted you!

Achievements

  • You have less than 500 followers. Just gave you a gift to help you succeed!
  • This is your first accepted contribution here in Utopian. Welcome!

Community-Driven Witness!

I am the first and only Steem Community-Driven Witness. Participate on Discord. Lets GROW TOGETHER!

mooncryption-utopian-witness-gif

Up-vote this comment to grow my power and help Open Source contributions like this one. Want to chat? Join me on Discord https://discord.gg/Pc8HG9x

Coin Marketplace

STEEM 0.18
TRX 0.15
JST 0.029
BTC 62153.21
ETH 2411.09
USDT 1.00
SBD 2.64