RE: Ingestron 0.0.2 released for every platform imaginable
Maybe I was a bit tired last night, but after looking at this again, I think I may see part of the problem, starting from when you wrote:
Does it need to use a websocket or can pubsub be achieved by other means?
For example, you talk about "saving from latency" and a "fully native" API. From the added context, respectfully, I think those things may mean something very different for me (and for the steemit devs) than they do for you. For example, if you're concerned about latency, clean code, etc, why would you be thinking of adding extra layers of external dependencies and complexity such as grpc?
And if you're already doing pull requests on their code while thinking on a completely different level than they are, well, put it this way. I wouldn't be comfortable touching their code without being pretty darn sure I know it inside and out (short of some very obvious bug fixes).
From your own code, you see that you are already talking to steemd with a websocket! From my perspective, the steemd is already easily accessible from ANY language (and over the net via wss://). It uses json-rpc. It doesn't get much simpler than that! It would be nice, though, to be able to do something like:
'{"jsonrpc": "2.0", "method": "call", "params": ["database_api", "sub_order_book", [10,protobuf]], "id": 3}'
to open a stream of order book updates, in json or even better, maybe as a protobuf (to save bandwidth).
In fact, this describes what I was saying perfectly, though I'm not sure why the decision was made to remove asynchronous notifications (which to me means subscriptions):
Websockets are provided by steemd because they're what we use to implement steemit.com. The steemit.com website is essentially an in-browser JavaScript implementation of most of the cli_wallet functionality (with lots of additional functionality and a pretty GUI as well, of course). Websockets are much saner to use for modern interactive web applications than HTTP connections. In BitShares, the previous Graphene-powered blockchain, websockets were also used for the server to send asynchronous notifications to the client, but Steem currently has little to no functionality that uses websockets in that way.
https://steemit.com/theoretical/@theoretical/how-to-use-the-steem-api
And regarding @officialfuzzy's "promises", I thought he meant this (but perhaps I missed an intended "double entendre" lol):
https://scotch.io/tutorials/javascript-promises-for-dummies
As for latency, you should also read @dantheman's recent post on how he's thinking about "latency":
https://steemit.com/blockchain/@dantheman/blockchains-should-be-designed-like-massively-multiplayer-games
Ingestron isn't here to save you from latency anymore. Ingestron is here to save you from API design these days. Originally ingestron used RethinkDB and was designed to enable real-time updates of page elements that had to be fetched from steemd.
No, fuzzy is talking about an application that i am shamefully overdue in delivering to him, for reasons that are my fault and reasons that aren't my fault, both.
Ingestron is a part of getting that done properly. Tell you what alex, I will make another post that explains how ingestron would change how people build apps using steem. So, stay tuned. It'll be complete with a diagram.
Dan's recent post was awesome. He's got a lot of brain in that brain pan of his, for sure!
also, AFAIK, steemd doesn't follow JSON-RPC 2.0, though it claims to. In fact, one can just look at the dependencies for ingestron to see this: Need to go into gorilla/rpc (a completley fine implementation of standard json-rpc 2.0.... and break it.
EG: Steem doesn't follow even the standards that it claims to follow. This sucks for devs, so an API that carefully implements a Swagger API will make the situation much better.
you really need to understand what's going on at a deeper level. if you're having problems with a missing "2.0" in the API, well, that goes to what I said about strict frameworks wanting everything a certain way. The real world's a lot more messy. There should be exception handling for that, and it could just make an assumption as to the version. If you've seen as many different APIs as I have, you'd better understand what I mean.
It may also help to spend some more time in chrome inspect and better understand how both ends talk when you perform an action. Steemit posts feedback in the console for every action it takes, which makes it much easier for people to understand how an actual implementation works.