Beem: Threaded blockchain.stream() fetched blocks without ops twice
Project Information
- Repository: https://github.com/holgern/beem
- Project Name: Beem
- Publisher: @holger80
A GitHub issue was created and the bug was confirmed and fixed by the PO.
Expected behavior
Blockchain.stream(threading=True)
should use threads to fetch several blocks in parallel. Each block should be fetched once from the RPC node.
Actual behavior
Blockchain.stream(threading=True)
requests blocks twice from the RPC node, if those block don't contain operations or only virtual operations. This limits the benefits from the parallel threads.
How to reproduce
from beem.blockchain import Blockchain
b = Blockchain()
for op in b.stream(start=2500000, stop=2500010, threading=True):
continue
Plus printf-debugging:
index 5cb196c..2e19545 100644
--- a/beemapi/steemnoderpc.py
+++ b/beemapi/steemnoderpc.py
@@ -53,6 +53,7 @@ class SteemNodeRPC(GrapheneRPC):
:raises ValueError: if the server does not respond in proper JSON format
:raises RPCError: if the server returns an error
"""
+ print(payload)
if self.url is None:
raise except
Output:
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_dynamic_global_properties', []], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': ['database_api', 'get_feed_history', []], 'jsonrpc': '2.0', 'id': 3}
{'method': 'call', 'params': ['database_api', 'get_feed_history', []], 'jsonrpc': '2.0', 'id': 4}
{'method': 'call', 'params': ['database_api', 'get_next_scheduled_hardfork', []], 'jsonrpc': '2.0', 'id': 5}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 6}
{'method': 'call', 'params': ['database_api', 'get_witness_schedule', []], 'jsonrpc': '2.0', 'id': 7}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 8}
{'method': 'call', 'params': ['database_api', 'get_reward_fund', ['post']], 'jsonrpc': '2.0', 'id': 9}
{'method': 'call', 'params': ['database_api', 'get_dynamic_global_properties', []], 'jsonrpc': '2.0', 'id': 10}
{'method': 'call', 'params': [None, 'get_block', [25031061]], 'jsonrpc': '2.0', 'id': 11}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': [None, 'get_block', [2500000]], 'jsonrpc': '2.0', 'id': 12}
{'method': 'call', 'params': [None, 'get_block', [2500001]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500002]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500003]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500004]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500005]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500006]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500007]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500000]], 'jsonrpc': '2.0', 'id': 13}
{'method': 'call', 'params': [None, 'get_block', [2500003]], 'jsonrpc': '2.0', 'id': 14}
{'method': 'call', 'params': [None, 'get_block', [2500004]], 'jsonrpc': '2.0', 'id': 15}
{'method': 'call', 'params': [None, 'get_block', [2500005]], 'jsonrpc': '2.0', 'id': 16}
{'method': 'call', 'params': [None, 'get_block', [2500007]], 'jsonrpc': '2.0', 'id': 17}
{'method': 'call', 'params': [None, 'get_block', [2500008]], 'jsonrpc': '2.0', 'id': 18}
{'method': 'call', 'params': [None, 'get_block', [2500009]], 'jsonrpc': '2.0', 'id': 3}
{'method': 'call', 'params': [None, 'get_block', [2500010]], 'jsonrpc': '2.0', 'id': 3}
{'method': 'call', 'params': [None, 'get_block', [2500008]], 'jsonrpc': '2.0', 'id': 19}
{'method': 'call', 'params': [None, 'get_block', [2500009]], 'jsonrpc': '2.0', 'id': 20}
{'method': 'call', 'params': [None, 'get_block', [2500010]], 'jsonrpc': '2.0', 'id': 21}
8 of the 11 blocks are fetched twice: 2500000, 2500003, 2500004, 2500005, 2500007, 2500008, 2500009, 2500010. Those blocks contain no operations.
The reason was confirmed to be here:
https://github.com/holgern/beem/blob/78281e724248b44724f985f8b57f936558d0d769/beem/blockchain.py#L463
with len(b.operations) == 0
, the block is treated as if it was never fetched from the chain.
The check for len(b.operations) == 0
was removed by the PO.
Environment
# beempy --version
beempy, version 0.19.53
# python --version
Python 3.6.6
Hi @stmdev, thanks for making this contribution.
I see the issue has been fixed, thanks to you pointing what exactly to look for.
Feedback:
This adds significant value to the open source community, I look forward to your next contribution.
Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post, click here.
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]
Thank you for your review, @fego!
So far this week you've reviewed 1 contributions. Keep up the good work!
Hi @stmdev! We are @steem-ua, a new Steem dApp, computing UserAuthority for all accounts on Steem. We are currently in test mode upvoting quality Utopian-io contributions! Nice work!
Hey @stmdev
Thanks for contributing on Utopian.
Congratulations! Your contribution was Staff Picked to receive a maximum vote for the bug-hunting category on Utopian for being of significant value to the project and the open source community.
We’re already looking forward to your next contribution!
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!