Introducing "App Stream": Curate and serve Steem posts/comments/replies by app/author/category
Hello Steem, today I will be outdooring a new Steem utility I have been working on: App Stream.
During the development of Peer Query beta as a full fledged platform featuring content created only on it, we run into a challenge.
While we could index posts and comments made on our site, we could not index comments/replies made from other clients on posts created through Peer Query.
This problem applies not only to Peer Query, but also to all other Steem clients.
Steem is great for content publishing, yet its open source data nature means that the same content can be served and interacted with from over several different interface.
This feature is good for open source and censorship resistance, however it poses a challenge for app/client owners who want to index all activities on posts made from their clients.
This is because while a post may be created on a one client, it would be edited, commented on, voted and resteemed from other clients.
Consider this challenge:
- Ama creates a query/post through PeerQuery.com
- John comments on it through raw Steem JS
- Jin replies to John's comment using Busy.org
- then the author Ama, edits the same post using Steemit.com
- Mary replies to Jin's reply using eSteem app
- countless more activities(resteem, votes, ...) happen on the same post from different clients
No matter how much indexing Peer Query does for activity on its site, we would definitely miss those made from other clients.
The only way to solve the problem is to stream all operation on the Steem blockchain and check if they match something in your DB or not.
App Stream is designed for this challenging task, and is designed to be as modular and efficient as possible. App Stream is a standalone Node.js program that curates all posts as well as its comments, replies and even edits from whatever target you set in a MYSQL DB.
There are 3 types of target:
There are six Db engines:
Countless other features and configuration instances, see Docs for more.
How to install
- Create empty DB, make sure MYSQL version is 5.7+ for
.envfile in root folder and enter DB details. See sample in
- Git clone
- Install dependencies:
- For custom setup, edit:
- Run system through setup manager:
- Use MYSQL Workbeanch/PHPMyAdmin to explore data in DB or visit
You do not need to create any tables for the DB. App Stream runs on:
- 2 tables with all kinds of fields: json, text, timestamp, varchar, auto-increment, ...
- 1 view of the each of the two tables above
- 3 complex conditional db indexing stored procedures each with 11 IN variables and 1 OUT variable
- 10 db api stored procedures with IN and OUT variables
Manual setup would be inefficient and difficult, and yet it has to be setup successfully before the main server is initialized.
db_setup module handles all this automatically for you, so you do not have to touch the DB at all. After all is done successfully it will automatically call the server.
Who can use App Stream?
Curating by app was App Stream's primary purpose, Steem clients would benefit the most from App Stream as they can use it to curate all posts made from their clients.
app_match option, you could curate data from only a specific version of an app, eg:
With a separate App Stream server, clients could server all content directly from their DB instead of relying on fetching content directly from RPC node endpoints.
App/client owners now can index all posts, comments, replies as well as edits made to posts made from their client, no matter which thirdparty client the activity originated from.
Curators would also greatly benefit from App Stream by using any of the category indexing engines:
We're working on Curator, a revolutionary open source interface for curating content which perfectly integrates into App Stream. Stay tuned.
I don't know if this is a thing yet, but with the post-comment-reply-by-author engine you can setup this system to curate all posts by an author, as well as the comments and replies by it.
While this is not recommended because Steem API allow this function through
steem.api.getDiscussionsByAuthorBeforeDate(), with App Stream's
source_app you can get specialized curation such as curating the author's post only if they are from a specific client such as Steemit.com, Busy.org, ...
With App Stream, you can curate all posts, comments and replies from any Steem author, category or app.
Such data would allow you to gain specialized data for specialized analysis.
App Stream comes with a painfully long and detailed documentation. You do not need to read it all to get App Stream running, its only there to explain everything about App Stream with examples incase you need to tweak it or get lost getting something to work.
See the documentation on the Github repo or via this link: https://github.com/peerquery/app-stream/blob/master/README.md.
Hopefully the detailed Table of Contents would make it easier to navigate.
- Depth determiner for each post, comment and reply - will accurately add depth to a reply whose dept is 37!
- API documentation on API endpoint bases
- Subdomain support
See Docs for more info with examples.
- Jailbreak to stream all Steem posts, comments and relies regardless of the source
- Host App Stream as a service and serve content via API
- Integrate with Hivemind for richer data interface
- Add a notifications feature
Curating only for the purposes of indexing in not enough. For clients such as Peer Query, we will need a full curation functionality.
I am already half way done with Curator, a powerful open source content curation interface for Steem. Curator will integrate with App Steam and allow a team(of multiple admins, moderator and curators) to curate content from Steem.
Curator comes with a full bot support which means all you have to do is supply the bot's account and posting key - it will periodically upvote all contents approved by the curators.
Like App Stream, Curator was initially being designed for Peer Query, however I realized that most of the content curators on Steem do so without a standalone app interface.
Then came the need to design Curator as a standalone app so other content curators(by tag, app or author) can also use it.
When Curator is done hopefully before this moth ends, Peer Query beta would be live; a complete move away from current query tag curation into a full platform with users, reputation, projects, curators and bots and more.
Thanks to @smooth and @howo for their recommendations on how to structure the db indexing functionality. Templates for the two streamer apps are by @almost-digital and @howo.
Thanks to you for reading, I hope you find App Stream of use.
Github repo: https://github.com/peerquery/app-stream
Thats a good idea! Ill have to check it out looks cool
Are you planning to unleash a web version? A local machine installation is too much of a bother for me and most other people, I reckon. Besides, not everyone can do it even provided the instructions, lol,
Thanks for asking - App Stream is a Node.js app that anyone can install anywhere: locally or on some cloud servers.
Let me put it this way, are you gonna host it somewhere? Like, make the official site for that?
Maybe this tool will help many people especially app owner save a lot of time.
Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!
Contributing on Utopian
Learn how to contribute on our website or by watching this tutorial on Youtube.
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!
I really loved the amount of work you have put into the contribution.
I also liked the presentation of the post.
I would love to recommend you to use JSDoc or some auto documentator to make a simple webpage which makes this documentation more presentable and attractive as devs prefer to read
READMEbut some prefer to see a simple webpage as a documentation.
I have staff picked this contribution because I believe its really high quality.
Thanks for providing such contributions.
Your contribution has been evaluated according to Utopian rules and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post,Click here
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
Introducing "App Stream": Curate and serve Steem posts/comments/replies by app/author/category has been resteemed by @resteemmuse
I agree with you apps are d best, but you know apps though can easily get deleted
who can delete the apps?
As in maybe a virus attack or something like a force factory set.
Though I still agree with you, irrespective of any short coming APPS is still the best choice so far
Hi @dzivenu! You have received 1.8 SBD @tipU upvote from @cardboard !
Earn daily income on steem: @tipU distributes 100% profit and additional 60% curation rewards to all investors and allows to automatically reinvest selected part of your payout.
Fanastic post friend
app steem is a great idea.... in app system there will be no posibility to hacking.....
wait for the readiness of the application, hopefully easier to help in interacting in steemit with this application, hopefully quickly resolved, so we will quickly use it
thanks for your interest, its already ready and you can use it right away.
Well, hope it helps me
Congratulation dzivenu! Your post has appeared on the hot page after 57min with 43 votes.
Thanks to @souldelas.