Hello Steem, today I will be outdooring a new Steem utility I have been working on: App Stream.
During the development of Peer Query beta as a full fledged platform featuring content created only on it, we run into a challenge.
While we could index posts and comments made on our site, we could not index comments/replies made from other clients on posts created through Peer Query.
This problem applies not only to Peer Query, but also to all other Steem clients.
Steem is great for content publishing, yet its open source data nature means that the same content can be served and interacted with from over several different interface.
This feature is good for open source and censorship resistance, however it poses a challenge for app/client owners who want to index all activities on posts made from their clients.
This is because while a post may be created on a one client, it would be edited, commented on, voted and resteemed from other clients.
Consider this challenge:
- Ama creates a query/post through PeerQuery.com
- John comments on it through raw Steem JS
- Jin replies to John's comment using Busy.org
- then the author Ama, edits the same post using Steemit.com
- Mary replies to Jin's reply using eSteem app
- countless more activities(resteem, votes, ...) happen on the same post from different clients
No matter how much indexing Peer Query does for activity on its site, we would definitely miss those made from other clients.
The only way to solve the problem is to stream all operation on the Steem blockchain and check if they match something in your DB or not.
App Stream is designed for this challenging task, and is designed to be as modular and efficient as possible. App Stream is a standalone Node.js program that curates all posts as well as its comments, replies and even edits from whatever target you set in a MYSQL DB.
There are 3 types of target:
There are six Db engines:
Countless other features and configuration instances, see Docs for more.
How to install
- Create empty DB, make sure MYSQL version is 5.7+ for
.envfile in root folder and enter DB details. See sample in
- Git clone
- Install dependencies:
- For custom setup, edit:
- Run system through setup manager:
- Use MYSQL Workbeanch/PHPMyAdmin to explore data in DB or visit
You do not need to create any tables for the DB. App Stream runs on:
- 2 tables with all kinds of fields: json, text, timestamp, varchar, auto-increment, ...
- 1 view of the each of the two tables above
- 3 complex conditional db indexing stored procedures each with 11 IN variables and 1 OUT variable
- 10 db api stored procedures with IN and OUT variables
Manual setup would be inefficient and difficult, and yet it has to be setup successfully before the main server is initialized.
db_setup module handles all this automatically for you, so you do not have to touch the DB at all. After all is done successfully it will automatically call the server.
Who can use App Stream?
Curating by app was App Stream's primary purpose, Steem clients would benefit the most from App Stream as they can use it to curate all posts made from their clients.
app_match option, you could curate data from only a specific version of an app, eg:
With a separate App Stream server, clients could server all content directly from their DB instead of relying on fetching content directly from RPC node endpoints.
App/client owners now can index all posts, comments, replies as well as edits made to posts made from their client, no matter which thirdparty client the activity originated from.
Curators would also greatly benefit from App Stream by using any of the category indexing engines:
We're working on Curator, a revolutionary open source interface for curating content which perfectly integrates into App Stream. Stay tuned.
I don't know if this is a thing yet, but with the post-comment-reply-by-author engine you can setup this system to curate all posts by an author, as well as the comments and replies by it.
While this is not recommended because Steem API allow this function through
steem.api.getDiscussionsByAuthorBeforeDate(), with App Stream's
source_app you can get specialized curation such as curating the author's post only if they are from a specific client such as Steemit.com, Busy.org, ...
With App Stream, you can curate all posts, comments and replies from any Steem author, category or app.
Such data would allow you to gain specialized data for specialized analysis.
App Stream comes with a painfully long and detailed documentation. You do not need to read it all to get App Stream running, its only there to explain everything about App Stream with examples incase you need to tweak it or get lost getting something to work.
See the documentation on the Github repo or via this link: https://github.com/peerquery/app-stream/blob/master/README.md.
Hopefully the detailed Table of Contents would make it easier to navigate.
- Depth determiner for each post, comment and reply - will accurately add depth to a reply whose dept is 37!
- API documentation on API endpoint bases
- Subdomain support
See Docs for more info with examples.
- Jailbreak to stream all Steem posts, comments and relies regardless of the source
- Host App Stream as a service and serve content via API
- Integrate with Hivemind for richer data interface
- Add a notifications feature
Curating only for the purposes of indexing in not enough. For clients such as Peer Query, we will need a full curation functionality.
I am already half way done with Curator, a powerful open source content curation interface for Steem. Curator will integrate with App Steam and allow a team(of multiple admins, moderator and curators) to curate content from Steem.
Curator comes with a full bot support which means all you have to do is supply the bot's account and posting key - it will periodically upvote all contents approved by the curators.
Like App Stream, Curator was initially being designed for Peer Query, however I realized that most of the content curators on Steem do so without a standalone app interface.
Then came the need to design Curator as a standalone app so other content curators(by tag, app or author) can also use it.
When Curator is done hopefully before this moth ends, Peer Query beta would be live; a complete move away from current query tag curation into a full platform with users, reputation, projects, curators and bots and more.
Thanks to you for reading, I hope you find App Stream of use.
Github repo: https://github.com/peerquery/app-stream