Introducing "App Stream": Curate and serve Steem posts/comments/replies by app/author/category

in utopian-io •  6 months ago


Hello Steem, today I will be outdooring a new Steem utility I have been working on: App Stream.

During the development of Peer Query beta as a full fledged platform featuring content created only on it, we run into a challenge.

While we could index posts and comments made on our site, we could not index comments/replies made from other clients on posts created through Peer Query.

This problem applies not only to Peer Query, but also to all other Steem clients.

The challenge

Steem is great for content publishing, yet its open source data nature means that the same content can be served and interacted with from over several different interface.

This feature is good for open source and censorship resistance, however it poses a challenge for app/client owners who want to index all activities on posts made from their clients.

This is because while a post may be created on a one client, it would be edited, commented on, voted and resteemed from other clients.

Consider this challenge:

  • Ama creates a query/post through PeerQuery.com
  • John comments on it through raw Steem JS
  • Jin replies to John's comment using Busy.org
  • then the author Ama, edits the same post using Steemit.com
  • Mary replies to Jin's reply using eSteem app
  • countless more activities(resteem, votes, ...) happen on the same post from different clients

No matter how much indexing Peer Query does for activity on its site, we would definitely miss those made from other clients.

The only way to solve the problem is to stream all operation on the Steem blockchain and check if they match something in your DB or not.

App Stream

App Stream is designed for this challenging task, and is designed to be as modular and efficient as possible. App Stream is a standalone Node.js program that curates all posts as well as its comments, replies and even edits from whatever target you set in a MYSQL DB.

There are 3 types of target:

  1. app
  2. category
  3. author

There are six Db engines:

  • post-by-app
  • post-by-author
  • post-by-category
  • post-comment-by-app
  • post-comment-by-author
  • post-comment-by-category
  • post-comment-reply-by-app
  • post-comment-reply-by-author
  • post-comment-reply-by-category

Countless other features and configuration instances, see Docs for more.

How to install

  1. Create empty DB, make sure MYSQL version is 5.7+ for json support
  2. Create .env file in root folder and enter DB details. See sample in docs/sample.env
  3. Git clone ssh:git@github.com:peerquery/app-stream.git or https://github.com/peerquery/app-stream.git
  4. Install dependencies: npm install
  5. For custom setup, edit: /config/config.js
  6. Run system through setup manager: node setup
  7. Use MYSQL Workbeanch/PHPMyAdmin to explore data in DB or visit api.localhost/api

Note:

You do not need to create any tables for the DB. App Stream runs on:

  • 2 tables with all kinds of fields: json, text, timestamp, varchar, auto-increment, ...
  • 1 view of the each of the two tables above
  • 3 complex conditional db indexing stored procedures each with 11 IN variables and 1 OUT variable
  • 10 db api stored procedures with IN and OUT variables

Manual setup would be inefficient and difficult, and yet it has to be setup successfully before the main server is initialized.

The db_setup module handles all this automatically for you, so you do not have to touch the DB at all. After all is done successfully it will automatically call the server.

Who can use App Stream?

Steem clients

Curating by app was App Stream's primary purpose, Steem clients would benefit the most from App Stream as they can use it to curate all posts made from their clients.

With the app_match option, you could curate data from only a specific version of an app, eg:   esteem/1.6.0,  esteem/1.5.1, or esteem/whatever.

With a separate App Stream server, clients could server all content directly from their DB instead of relying on fetching content directly from RPC node endpoints.

App/client owners now can index all posts, comments, replies as well as edits made to posts made from their client, no matter which thirdparty client the activity originated from.

Steem Curators

Curators would also greatly benefit from App Stream by using any of the category indexing engines:

  • post-by-category
  • post-comment-by-category
  • post-comment-reply-by-category

We're working on Curator, a revolutionary open source interface for curating content which perfectly integrates into App Stream. Stay tuned.

Steem blogs

I don't know if this is a thing yet, but with the post-comment-reply-by-author engine you can setup this system to curate all posts by an author, as well as the comments and replies by it.

While this is not recommended because Steem API allow this function through  steem.api.getDiscussionsByAuthorBeforeDate(), with App Stream's source_app you can get specialized curation such as curating the author's post only if they are from a specific client such as Steemit.com, Busy.org, ...

Steem BI

With App Stream, you can curate all posts, comments and replies from any Steem author, category or app.

Such data would allow you to gain specialized data for specialized analysis.

Documentation

App Stream comes with a painfully long and detailed documentation. You do not need to read it all to get App Stream running, its only there to explain everything about App Stream with examples incase you need to tweak it or get lost getting something to work.

See the documentation on the Github repo or via this link: https://github.com/peerquery/app-stream/blob/master/README.md.

Hopefully the detailed Table of Contents would make it easier to navigate.

Cool addons

  • Depth determiner for each post, comment and reply - will accurately add depth to a reply whose dept is 37!
  • API documentation on API endpoint bases
  • Subdomain support

See Docs for more info with examples.

Cool hacks

  • Jailbreak to stream all Steem posts, comments and relies regardless of the source
  • Host App Stream as a service and serve content via API
  • Integrate with Hivemind for richer data interface
  • Add a notifications feature

What's next

Curating only for the purposes of indexing in not enough. For clients such as Peer Query, we will need a full curation functionality.

I am already half way done with Curator, a powerful open source content curation interface for Steem. Curator will integrate with App Steam and allow a team(of multiple admins, moderator and curators) to curate content from Steem.

Curator comes with a full bot support which means all you have to do is supply the bot's account and posting key - it will periodically upvote all contents approved by the curators.

Like App Stream, Curator was initially being designed for Peer Query, however I realized that most of the content curators on Steem do so without a standalone app interface.

Then came the need to design Curator as a standalone app so other content curators(by tag, app or author) can also use it.

When Curator is done hopefully before this moth ends, Peer Query beta would be live; a complete move away from current query tag curation into a full platform with users, reputation, projects, curators and bots and more.

Acknowledgment

Thanks to @smooth and @howo for their recommendations on how to structure the db indexing functionality. Templates for the two streamer apps are by @almost-digital and @howo.

Thanks to you for reading, I hope you find App Stream of use.

Links

Github repo: https://github.com/peerquery/app-stream

Docs: https://github.com/peerquery/app-stream/blob/master/README.md

Author

https://github.com/Dzivenu


Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Thats a good idea! Ill have to check it out looks cool

Are you planning to unleash a web version? A local machine installation is too much of a bother for me and most other people, I reckon. Besides, not everyone can do it even provided the instructions, lol,

·

Thanks for asking - App Stream is a Node.js app that anyone can install anywhere: locally or on some cloud servers.

·
·

Let me put it this way, are you gonna host it somewhere? Like, make the official site for that?

Maybe this tool will help many people especially app owner save a lot of time.

Hey @dzivenu
Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!

Contributing on Utopian
Learn how to contribute on our website or by watching this tutorial on Youtube.

Want to chat? Join us on Discord https://discord.gg/h52nFrV.

Vote for Utopian Witness!

I really loved the amount of work you have put into the contribution.

I also liked the presentation of the post.

I would love to recommend you to use JSDoc or some auto documentator to make a simple webpage which makes this documentation more presentable and attractive as devs prefer to read README but some prefer to see a simple webpage as a documentation.

I have staff picked this contribution because I believe its really high quality.

Thanks for providing such contributions.

Keep contributing.

Your contribution has been evaluated according to Utopian rules and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post,Click here


Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]

·

thank you

I agree with you apps are d best, but you know apps though can easily get deleted

·

who can delete the apps?

·
·

As in maybe a virus attack or something like a force factory set.
Though I still agree with you, irrespective of any short coming APPS is still the best choice so far

Congratulation dzivenu! Your post has appeared on the hot page after 57min with 43 votes.
Thanks to @souldelas.

wait for the readiness of the application, hopefully easier to help in interacting in steemit with this application, hopefully quickly resolved, so we will quickly use it

·

thanks for your interest, its already ready and you can use it right away.

·
·

Well, hope it helps me

Fanastic post friend

app steem is a great idea.... in app system there will be no posibility to hacking.....

Introducing "App Stream": Curate and serve Steem posts/comments/replies by app/author/category has been resteemed by @resteemmuse