Python steem API not fast enough to keep up with new blocks
I've been having some issues with my @croupierbot bot recently. At first, everything seemed to be working using the stream_comments method in a loop. Well, working? I needed to run everything in a while loop with a try/except, catching steembase.exceptions.PostDoesNotExist exceptions and restarting the stream_comments loop all the time, but after that, my beta tests did OK. For a while that is.
Then, after @bigpanda tried to use my bot, I noticed something was off. @croupierbot missed many of the times its name got mentioned in the stream. It looked like this was getting worse over time in fact. And apart from that, the time between calling in @croupierbot and getting the first response often went up as the bot was left running for a larger amount of time.
So I thought, hey, maybe it is the exception stuff, maybe the restarting of the stream API had me skipping stuff, let's write to the get_blocks_range API instead.
The results weren't good. No more exceptions to catch, but the timelapse problem went completely off the scale. So what to do. I stripped down my script and removed all the stuff that really did something. I ended up with a little script that tried to keep up with new blocks being added to the blockchain. Only that, nothing more. And guess what? The script couldn't even keep up with doing nothing. That is, it seems the Python steem API is too slow to allow for writing a bot. Silly, right? I must be something wrong here, right? Well, I thought so, but after looking at my script for hours, and I really can't see anything stupid. Here is my code:
And here is the result of letting the code run for a number of minutes:
Messed up, right? In the time it takes to request ten blocks from the steem API, more often than not more than ten new blocks get added to the blockchain! Am I doing something really stupid here that isn't obvious to me right now? Or has the steemit network become too busy for the API to keep up with new block creation?
For now, it seems I will need to find a different strategy for @resteembot. I hate the idea of asking for transactions, even if they are $ 0.001 SBD rather than scraping for mentions as I was doing, but it is possible that currently, it is my only option to get @croupierbot to work relyably. If you spot anything stupid in my code or know a better coding strategy for scrapping for mentions, please let me know in the comments.
Tnx.
Hi again @pibara!
I'm super happy I found this piece of code, because it made me dig deeper and compare different methods to pull data from the blockchain. Maybe you have already progressed with this, but if you haven't, I have to warn you. ;) I found an extremely simple error. You wanna pull your hair for sure.
I couldn't believe it was going to be this slow. At least I expected this to be faster than using the stream_comment functions. I replicated your script and after some testing with different RPC nodes and some other tweaking, I realised that the starting block number didn't increase. For each iteration, it started over from the initial value.
Then I realised that this line was put inside the while-loop!!
index = startIn my tests now, I have much better success using the get_blocks_range rather than stream_comments.
So thanks again for sharing this piece of test code!
Thanks, seems the biggest speed problem was mostly with the standard server used by the API.
Moving to the Blockchain based stream_from now, hope to get a fully working and restartable version up this weekend based on that. Basically base loop will be around:
s=steem.steemd.Steemd(["https://gtg.steem.house:8090","https://seed.bitcoiner.me"])
bs = steem.blockchain.Blockchain(s)
for entry in bs.stream_from(p.state["block_no"]):
...
Makes the difference between running half an hour extra behind each hour and catching up over two whole days each hour.
I wish I could read code and understand the program here, but I really think @croupierbot can be something incredible. I'll keep trying it as long as you want me to @pibara. Just let me know whenever you want me to give it a trial run and I will.
Do you maybe know who created the originalworks bot? That bot doesn't seem to have any of the API issues I'm running into, so maybe its author might be able to give me some pointers into the right direction.
@unprovoked wrote it , check with him .Tho i think he used piston
Hi, @bigpanda
Thank you for inviting me to this page.
I am your friendly incorruptible croupier.
I am sorry for the inconvenience, but 'can' is not a command I currently understand.
[@croupierbot catch-up instance]
@croupierbot lottery
Hi, @pibara
Thank you for inviting me to your page.
I am your friendly incorruptible croupier.
I am sorry for the inconvenience, but the post you have invited me on currently has too few upvoted comments for me to run a lottery with.
Please invite me again after you have gathered at least two contenders on this page.
@croupierbot whatsup
Hi, @pibara
Thank you for inviting me to this page.
I am your friendly incorruptible croupier.
I am sorry for the inconvenience, but 'whatsup' is not a command I currently understand.
@croupierbot ping