You are viewing a single comment's thread from:
RE: Steemwallet.com: How to improve security in a least authority way AND how "NOT" to roll-out such security measures.
I agree with many things in your article, but c'mon, if you use an rpc heavy script, deploy your own node. Nodes are expensive and steemit rightfully limited their api so that their funds are not wasted on unoptimized bots.
For example buildteam runs their own private nodes that costs about 1k per month or more. We use rpc nodes a lot. We pay for our use.
If your stats gain you money or are worth it, use that money to pay for rpc nodes. Same as you pay for the computer doing the calculation.
What my script did was:
The result would then get processed into something like this.
The stats were made more in support of the work being done by the @freezepeach initiative than for making money. Most posts made just a few cents, half of what I had set @freezepeach as beneficiary for, so that wasn't the point. The point was making people who were being structurally targeted by high powered flagging for disputable reasons to be made aware of the @freezepeach initiative.
Like most stats script, there would be a large amount of API calls done in a short time and the rest of the day zero. There are reasons to do rate limiting, proper rate limiting with window sizes, flow control, etc, you likely know the routine. That however isn't hos it was implemented. They just made up a number and a timeframe. Hit the number of API calls they had set as upper limit within 5 minutes and get 4xx errors for a while after that. No library, including my own is capable of working that into a decent client side rate limit implementation that way.
So it is not the rate limit itself that is the problem, just like it wasn't the HF20 features that were the problem, it is the implementation and lack of adoption requirements that could allow for gracefull degradation of the service instead of a disruptive one.
Hope I'm at least partially making sense.
Hmmm if you do something like that, maybe try steemsql? That can run aggregate queries.
Allowing such bursts would allow apps to basically DDOS the api, especially with the expensive history lookups you do.
If you want to improve your design, stream blocks, then don't collect posts but flags and then put them in a queue and query one of these posts per 3 seconds or something. And at the end of the day do one query of the data you collected over the day.
No real world api allows such expensive bursts without limiting you very quickly.
Possibly valid for my script, but not really the point why I brought up the rate limit example.
Other API's that implement rate limits tend to use custom HTTP headers like:
Or simply:
As a library author, these headers are something you can work with to implement proper client side rate limiting. You can run statistics scripts that run at exactly the server defined rate-limit speed. That means a script that used to run for an hour might run for 15 hours now, but it will still run. Graceful degradation.
There are established best practices for implementing rate limiting for a web API. Just picking a limit size and period that the client would normally be unable to guess and then start responding with 4xx errors for a period of time the client couldn't reasonably guess either is in no way graceful degradation. It is a disruptive roll-out.
Yeah I can agree with that. They probably needed something that stops the flood quickly and did not do any play nice stuff.