I'm Mr. Meeseeks! Look at me!

in utopian-io •  2 months ago

Redis based block follower is an efficient way for multiple apps to stream the Steem Blockchain.

If you have multiple applications that need to perform actions as operations occur, meeseeker will allow your apps to each perform actions for specific operations without each app having to stream the entire blockchain.

In a nutshell: The overarching intent here is to provide a "live view" of the blockchain, not store the entire blockchain. Apps can attach to your redis source and ask, "What just happened?"

Repository:

https://github.com/inertia186/meeseeker

Purpose

Although Meeseeker tracks all operations, it is only intended to provide other applications signals that those operations have happened. It is not intended to provide cryptographically verifiable events.

Possible uses:

  • Notifications of events, suitable for push to mobile devices or web browsers.
  • Invoke periodic updates on a threshold.
  • Light-weight bots that only care about a limit set of operations, reducing the number of API calls.

Why Redis?

Redis is a persistent key-value database, with built-in net interface. See: https://redis.io/

It allows for quick storage and lookup of operations by key as well as the ability to automatically expire keys that are no longer needed.

Installation

First, install redis:

On linux:

sudo apt install redis-server

On macOS:

brew install redis

Next, install ruby. One way to do this is install rvm. Once ruby is installed, install meeseeker with the gem command:

gem install meeseeker

This installs meeseeker as a command available to the OS, e.g.:

meeseeker help

To do the actual sync to your local redis source (defaults assume redis://127.0.0.1:6379/0):

meeseeker sync

To specify an alternative redis source:

MEESEEKER_REDIS_URL=redis://:p4ssw0rd@10.0.1.1:6380/15 meeseeker sync

You can also specify an alternative Steem node:

MEESEEKER_NODE_URL=https://api.steemit.com meeseeker sync

To sync from the head block instead of the last irreversible block:

MEESEEKER_STREAM_MODE=head meeseeker sync

To ignore virtual operations (useful if the node doesn't enable get_ops_in_blocks or if you want to sync from the head block):

MEESEEKER_INCLUDE_VIRTUAL=false meeseeker sync

Normally, keys stay on redis for 24 hours. If you want to change this behavior, use MEESEEKER_EXPIRE_KEYS and specify the new value in seconds, for example:

MEESEEKER_EXPIRE_KEYS=10 meeseeker sync

Usage

When meeseeker sync starts for the first time, it initializes from the last irreversible block number. If the sync is interrupted, it will resume from the last block sync'd unless that block is older than MEESEEKER_EXPIRE_KEYS in which case it will skip to the last irreversible block number.

Using SUBSCRIBE

For redis-cli, please see: https://redis.io/topics/pubsub

Channels available for meeseeker:

  • steem:block
  • steem:transaction
  • steem:op:vote
  • steem:op:comment
  • steem:op:whatever (replace "whatever" with the op you want)

As mentioned in the last whatever example, all operation types can be subscribed to as channels, including virtual operations, if enabled.

For example, from redis-cli, if we wanted to stream block numbers:

$ redis-cli
127.0.0.1:6379> subscribe steem:block
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "steem:block"
3) (integer) 1
1) "message"
2) "steem:block"
3) "{\"block_num\":29844374}"
1) "message"
2) "steem:block"
3) "{\"block_num\":29844375}"
1) "message"
2) "steem:block"
3) "{\"block_num\":29844376}"

A ruby application can subscribe to a channel as well, using the redis gem:

require 'redis'

url = 'redis://127.0.0.1:6379/0'
ctx = Redis.new(url: url)

Redis.new(url: url).subscribe('steem:op:comment') do |on|
  on.message do |channel, message|
    payload = JSON[message]
    comment = JSON[ctx.get(payload['key'])]
    
    puts comment['value']
  end
end

Many other clients are supported: https://redis.io/clients

Using SCAN

From the redis manual:

Since these commands allow for incremental iteration, returning only a small number of elements per call, they can be used in production without the downside of commands like KEYS or SMEMBERS that may block the server for a long time (even several seconds) when called against big collections of keys or elements.

However while blocking commands like SMEMBERS are able to provide all the elements that are part of a Set in a given moment, The SCAN family of commands only offer limited guarantees about the returned elements since the collection that we incrementally iterate can change during the iteration process.

See: https://redis.io/commands/scan

Keep in mind that SCAN requires pagination to get a complete result. Redis implements pagination using a cursor based iterator.

See: https://redis.io/commands/scan#scan-basic-usage

Once your sync has started, you can begin doing queries against redis, for example, in the redis-cli:

redis-cli --scan --pattern 'steem:*:vote'

This returns the keys, for example:

steem:29811083:7fd2ea1c73e6cc08ab6e24cf68e67ff19a05896a:0:vote
steem:29811085:091c3df76322ec7f0dc51a6ed526ff9a9f69869e:0:vote
steem:29811085:24bfc199501779b6c2be2370fab1785f58062c5a:0:vote
steem:29811086:36761db678fe89df48d2c5d11a23cdafe57b2476:0:vote
steem:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote
steem:29811085:44036fde09f20d91afda8fc2072b383935c0b615:0:vote
steem:29811086:570abf0fbeeeb0bb5c1e26281f0acb1daf175c39:0:vote
steem:29811083:e3ee518c4958a10f0d0c5ed39e3dc736048e8ec7:0:vote
steem:29811083:e06be9ade6758df59e179160b749d1ace3508044:0:vote

To get the actual vote operation for a particular key, use:

redis-cli get steem:29811085:f904ac2e5e338263b03b640a4d1ff2d5fd01169e:0:vote

If, on the other hand, you want custom_json only:

redis-cli --scan --pattern 'steem:*:custom_json'

This only returns the related keys, for example:

steem:29811084:43f1e1a367b97ea4e05fbd3a80a42146d97121a2:0:custom_json
steem:29811085:5795ff73234d64a11c1fb78edcae6f5570409d8e:0:custom_json
steem:29811083:2d6635a093243ef7a779f31a01adafe6db8c53c9:0:custom_json
steem:29811086:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:0:custom_json
steem:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json
steem:29811083:04a6ddc83a63d024b90ca13996101b83519ba8f5:0:custom_json

To get the actual custom json operation for a particular key, use:

redis-cli get steem:29811083:7fbbde120aef339511f5af1a499f62464fbf4118:0:custom_json

To get all transactions for a particular block number:

redis-cli --scan --pattern 'steem:29811085:*'

Or to get all ops for a particular transaction:

redis-cli --scan --pattern 'steem:*:31ecb9c85e9eabd7ca2460fdb4f3ce4a7ca6ec32:*'



See some of my previous Ruby How To posts in: #radiator #ruby

Get in touch!

If you're using Radiator, I'd love to hear from you. Drop me a line and tell me what you think! I'm @inertia on STEEM.

License

I don't believe in intellectual "property". If you do, consider Radiator as licensed under a Creative Commons CC0 License.

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  
Loading...

Existence is pain to a Meeseeks...



couldn't help myself.

I like this project though. Nice work.

·

You got it! I felt inspired by their notion that existence is pain. Storing ops off the blockchain is pain. This should help a little with that because, a) it only downloads ~400 MB once per day but shares the data for all apps, and b) expire the ops so they don't take up space over time.

So the ops only exist for 24 hours because existence is pain.

Redis is good for this because it provides a usable UI. I've done the same with kakfa, but there's no UI. It's just straight application integration.

·

Which visualizer do you use?

·
·

I visualize everything through shell or tools I wrote

·
·
·

I guess I don't understand what you mean by "a usable UI."

·
·
·
·

I mean for administration and management. Everything is over http/https. Out-of-the-box, you can just curl stuff in redis. Kafka, not so much :(

One huge advantage I just realized from redis is that most message queues and pubsub systems require you to build state management yourself. Redis has state-management builtin. It's possible to save your state to develop workflows based-on events much like AWS steps.

I'm only thinking about this because I'm working on ways to integrate steem-connect, identity management, and workflow system into steem using custom_json transactions. I'm working with AWS Neptune graph to manage non-linear workflows, but now I'm wondering if redis wouldn't just be easier, but also more localized. Imagine being able to build bots into the platform that can follow very complex instructions like community managed curation, software support, and integration with other platforms like github, discord, trello, Jira, basecamp, etc... None of things would be coupled or even aware of steem, but at the same time providing resources used in a workflow built on steem.

Interesting! Look at me!!

Posted using Partiko Android

Hi @inertia!

Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your UA account score is currently 6.574 which ranks you at #147 across all Steem accounts.
Your rank has improved 1 places in the last three days (old rank 148).

In our last Algorithmic Curation Round, consisting of 241 contributions, your post is ranked at #43.

Evaluation of your UA score:
  • You've built up a nice network.
  • The readers appreciate your great work!
  • Try to improve on your user engagement! The more interesting interaction in the comments of your post, the better!

Feel free to join our @steem-ua Discord server

This looks great. I have one use case I'm working on which is about monitoring transfers from and to a particular account. I'm using steem.js and there is a minute or two of delays. Can I use meeseeker to speed up things?

·

I could see it being one component of that solution.

Actually, maybe you’re existing solution needs to use the head block number to see the transfers more quickly, keeping in mind they are reversible. Although meeseeker supports streaming the head block, I’m not sure it will speed things up other than the fact that downstream components won’t have to monitor all operations.

Hey, @inertia!

Thanks for contributing on Utopian.
Congratulations! Your contribution was Staff Picked to receive a maximum vote for the development category on Utopian for being of significant value to the project and the open source community.

We’re already looking forward to your next contribution!

Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).

Want to chat? Join us on Discord https://discord.gg/h52nFrV.

Vote for Utopian Witness!

·

Plz dont low reting

·
·

Source
Spamming comments is frowned upon by the community. Continued comment spamming may result in action from the cheetah bot.

More Information:
The Art of Commenting
Comment Classifications