Developing Steem Files : Journal Entry for 2018 03(March) 08
March 8th, 2018
In this developer's journal of Steem Files:
- Steem Files dependencies
- Shifting Sand underneath
- An ALTERNATIVE to Steem-Python
- Getting Steem-Python to work with the new coming hard-fork
Steemfiles and it's dependencies
In writing Steemfiles, I needed to rely on libraries that would handle the calls to RPC. There is only so much a developer can do, so of course I use libraries. With the exception of system level embedded programmers we always do. I downloaded @xeroc's piston package and got started. I found I needed to upgrade Python to version 3. I opted to get the latest Python and I am really glad I did!
So, Steemfiles requires several things to work: Public RPC nodes, MySQL compatible server, a Steem account (@steemfiles), a web-server (Apache2). For the custom software to work, it needs python-steem, mysql-connector, and python 3.6. Now, if anything of these things change thier interfaces, then things would break. So, the policy of not upgrading anything gets adopted by many places. Then there is the problem of new security vulnerabilities that come out.
The sand shifts under the structure
Because Steem is relatively new and, when I started Steemfiles it was really new, one could expect some changes and certainly, the Steem developers have be open to many consensus changes.
Piston required an RPC node. Because there were few RPC nodes at the time, Steemfiles stopped working when the default RPC node no longer worked. More RPC nodes were tried and things worked fine for a while. Later, functions that belonged to Piston were exported to Steem-Python. Now a days, I have migrated everything to Steem-Python. Recently, version 1 of Steem Connect has stopped working and so I am pleased to say that Steem files no longer needs Steem Connect. Steem Files will run just fine if Steem Connect ceased to exist.
Appbase is coming.
Appbase, is the new API version of 19.4 HF. As a compatibility feature all of the old apis will work if you set the api parameter to 'condenser_api'. Now for compatibility, it would have been better to leave the things the way they are in 19.3. Steem-Python doesn't have API parameters in its API. It is something it hides from the users of its library. So, it is up to Steem-Python to adapt its implementation.
Instead of waiting though, I forked the project and added Appbase support.
Steem-Python vs. Beem
There is a cometitive API to Steem-Python called Beem. It has a slightly different API and already has Appbase support. It is an option to migrate to. Most of the time migration is for the birds, it might be a good idea to do so in this case. It also has 200 unit tests.
Porting Steem-Python to HF 19.4
I forked Steem-Python repository, and looked around to decide where changes should be made. I decided that the HttpClient class should be adapted. It already is doing fail-over in this class. For illustration purposes I have removed some lines of code for brevity:
def call(self,
name,
*args,
api=None,
return_with_args=None,
_ret_cnt=0,
kwargs=None):
...
body = HttpClient.json_rpc_body(name, *args, api=api, kwargs=kwargs)
response = None
try:
response = self.request(body=body)
except (MaxRetryError, ConnectionResetError, ReadTimeoutError,
RemoteDisconnected, ProtocolError) as e:
# handling for connection issues here
except Exception as e:
# bad node output handled here
else:
# This is if no exceptions occur
if response.status not in tuple([*response.REDIRECT_STATUSES,
200]):
logger.info('non 200 response:%s', response.status)
return self._return(
response=response,
args=args,
return_with_args=return_with_args)
Python gives us the option to do something if no exceptions were raised. that's what the else is for. Nice feature. In this the RPC node that uses HF 19.4 will return an error message: So, I changed the function above with this (again removed lines for brevity):
def call(self,
name,
*args,
api=None,
return_with_args=None,
_ret_cnt=0,
kwargs=None):
...
if _node_version is None:
_node_version = self.node_version
body = None
if _node_version is None or _node_version < 19.4:
body = HttpClient.json_rpc_body(name, *args, api=api, kwargs=kwargs)
else:
body = HttpClient.json_rpc_body(name, *args, api='condenser_api', kwargs=kwargs)
response = None
try:
response = self.request(body=body)
except (MaxRetryError, ConnectionResetError, ReadTimeoutError,
RemoteDisconnected, ProtocolError) as e:
# handling for connection issues here
except Exception as e:
# bad node output handled here
else:
# This is if no exceptions occur
if response.status not in tuple([*response.REDIRECT_STATUSES,
200]):
logger.info('non 200 response:%s', response.status)
answer = None
try:
answer = self._return(
response=response,
args=args,
return_with_args=return_with_args)
self.node_version = _node_version
except Exception as e:
# Default 19.3 didn't work. Try using 19.4
if self.node_version is None:
answer = self.call(
name,
*args,
return_with_args=return_with_args,
_ret_cnt=_ret_cnt + 1, _node_version=19.4)
return answer
There were other small changes, but this is the meat you could say. Now, we have three values for node_version:
None (which means we don't know), 19.3, and 19.4. When node_version is None we try both and when there is a non-error result from the server we set the version to one of these version numbers.
There is a self.node_version which is the recorded node version for the current node. There is also the parameter to call() _node_version which specifies what to try.
There really is no need to do anything else. I guess Steem-Python coders will eventually find their own ways to upgrade to 19.4 and it will probably be a more elegant solution but I wanted to be ready immediately for it!
Dash XjzjT4mr4f7T3E8G9jQQzozTgA2J1ehMkV
LTC LLXj1ZPQPaBA1LFtoU1Gkvu5ZrxYzeLGKt
BitcoinCash 1KVqnW7wZwn2cWbrXmSxsrzqYVC5Wj836u
Bitcoin 1Q1WX5gVPKxJKoQXF6pNNZmstWLR87ityw (too expensive to use for tips)
See also
See my other recent article:
@originalworks