Beem: Threaded blockchain.stream() fetched blocks without ops twice

Project Information

• Repository: https://github.com/holgern/beem
• Project Name: Beem
• Publisher: @holger80

A GitHub issue was created and the bug was confirmed and fixed by the PO.

Expected behavior

Blockchain.stream(threading=True) should use threads to fetch several blocks in parallel. Each block should be fetched once from the RPC node.

Actual behavior

Blockchain.stream(threading=True) requests blocks twice from the RPC node, if those block don't contain operations or only virtual operations. This limits the benefits from the parallel threads.

How to reproduce

from beem.blockchain import Blockchain
b = Blockchain()
for op in b.stream(start=2500000, stop=2500010, threading=True):
    continue

Plus printf-debugging:

index 5cb196c..2e19545 100644
--- a/beemapi/steemnoderpc.py
+++ b/beemapi/steemnoderpc.py
@@ -53,6 +53,7 @@ class SteemNodeRPC(GrapheneRPC):
             :raises ValueError: if the server does not respond in proper JSON format
             :raises RPCError: if the server returns an error
         """
+        print(payload)
         if self.url is None:
             raise except

Output:

{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_dynamic_global_properties', []], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': ['database_api', 'get_feed_history', []], 'jsonrpc': '2.0', 'id': 3}
{'method': 'call', 'params': ['database_api', 'get_feed_history', []], 'jsonrpc': '2.0', 'id': 4}
{'method': 'call', 'params': ['database_api', 'get_next_scheduled_hardfork', []], 'jsonrpc': '2.0', 'id': 5}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 6}
{'method': 'call', 'params': ['database_api', 'get_witness_schedule', []], 'jsonrpc': '2.0', 'id': 7}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 8}
{'method': 'call', 'params': ['database_api', 'get_reward_fund', ['post']], 'jsonrpc': '2.0', 'id': 9}
{'method': 'call', 'params': ['database_api', 'get_dynamic_global_properties', []], 'jsonrpc': '2.0', 'id': 10}
{'method': 'call', 'params': [None, 'get_block', [25031061]], 'jsonrpc': '2.0', 'id': 11}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': ['database_api', 'get_config', []], 'jsonrpc': '2.0', 'id': 1}
{'method': 'call', 'params': [None, 'get_block', [2500000]], 'jsonrpc': '2.0', 'id': 12}
{'method': 'call', 'params': [None, 'get_block', [2500001]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500002]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500003]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500004]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500005]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500006]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500007]], 'jsonrpc': '2.0', 'id': 2}
{'method': 'call', 'params': [None, 'get_block', [2500000]], 'jsonrpc': '2.0', 'id': 13}
{'method': 'call', 'params': [None, 'get_block', [2500003]], 'jsonrpc': '2.0', 'id': 14}
{'method': 'call', 'params': [None, 'get_block', [2500004]], 'jsonrpc': '2.0', 'id': 15}
{'method': 'call', 'params': [None, 'get_block', [2500005]], 'jsonrpc': '2.0', 'id': 16}
{'method': 'call', 'params': [None, 'get_block', [2500007]], 'jsonrpc': '2.0', 'id': 17}
{'method': 'call', 'params': [None, 'get_block', [2500008]], 'jsonrpc': '2.0', 'id': 18}
{'method': 'call', 'params': [None, 'get_block', [2500009]], 'jsonrpc': '2.0', 'id': 3}
{'method': 'call', 'params': [None, 'get_block', [2500010]], 'jsonrpc': '2.0', 'id': 3}
{'method': 'call', 'params': [None, 'get_block', [2500008]], 'jsonrpc': '2.0', 'id': 19}
{'method': 'call', 'params': [None, 'get_block', [2500009]], 'jsonrpc': '2.0', 'id': 20}
{'method': 'call', 'params': [None, 'get_block', [2500010]], 'jsonrpc': '2.0', 'id': 21}

8 of the 11 blocks are fetched twice: 2500000, 2500003, 2500004, 2500005, 2500007, 2500008, 2500009, 2500010. Those blocks contain no operations.

---

The reason was confirmed to be here:
https://github.com/holgern/beem/blob/78281e724248b44724f985f8b57f936558d0d769/beem/blockchain.py#L463
with len(b.operations) == 0, the block is treated as if it was never fetched from the chain.

The check for len(b.operations) == 0 was removed by the PO.

Environment

# beempy --version
beempy, version 0.19.53
# python --version
Python 3.6.6

GitHub Account

https://github.com/crokkon