I.T. Spices The LINUX Way

lightingmacsteem (55)in #blockchain • 6 years ago

Python In The Shell: The STEEMIT Ecosystem – Post #99

BLOCKCHAIN AND DATABASE MANIPULATIONS USING PYTHON

In continuation as per the previous post here:

https://steemit.com/blockchain/@lightingmacsteem/4jgp4m-i-t-spices-the-linux-way

It is time for us to go to the lines of codes that will manipulate the database and the JSON file that carries the steemit blockchain records for database insertion, said lines were the very ones being discussed as to how on the previous blogs.

The lines of codes below:

1 ###CHECK THE DESTINATION FILE
2 dd = ('sed -n "$=" ' + destfile)
3 checkfile = (os.popen(dd).read()).strip()
4 if (str(checkfile) != str(0)) and os.path.isfile(destfile):
5   
6   ###BE SURE TO OPEN FILE IN UTF-8
7   list = open(destfile, 'r', encoding='utf-8')
8   print('There is/are new blocks, I will update the database with about ' + checkfile + ' new records; please wait.......')
9   print('There is/are new blocks, I will update the database with about ' + checkfile + ' new records; please wait.......', file=open(logfile, 'a'))
10  time.sleep(9)
11
12  ###LOOP LINE PER LINE HERE USING WHILE
13  for line in list:
14      
15      ###WHOLE BLOCKDATA HERE UNALTERED
16      blockdata = str(line).rstrip()
17  
18      ###PYTHON
19      jason = json.loads(line)
20      if (jason["block_id"]) == []:
21          block_id = "NA"
22      else:
23          try:
24              block_id = (jason["block_id"])
25          except (OSError, TypeError):
26              block_id = "ERROR"
27      
28      if (jason["extensions"]) == []:
29          extensions = "NA"
30      else:
31          try:
32              extensions = (jason["extensions"])
33          except (OSError, TypeError):
34              extensions = "ERROR"
35      
36      if (jason["previous"]) == []:
37          previous = "NA"
38      else:
39          try:
40              previous = (jason["previous"])
41          except (OSError, TypeError):
42              previous = "ERROR"
43      
44      if (jason["signing_key"]) == []:
45          signing_key = "NA"
46      else:
47          try:
48              signing_key = (jason["signing_key"])
49          except (OSError, TypeError):
50              signing_key = "ERROR"
51      
52      if (jason["timestamp"]) == []:
53          timestamp = "NA"
54      else:
55          try:
56              timestamp = (jason["timestamp"])
57          except (OSError, TypeError):
58              timestamp = "ERROR"
59      
60      if (jason["transaction_merkle_root"]) == []:
61          transaction_merkle_root = "NA"
62      else:
63          try:
64              transaction_merkle_root = (jason["transaction_merkle_root"])
65          except (OSError, TypeError):
66              transaction_merkle_root = "ERROR"
67      
68      if (jason["witness"]) == []:
69          witness = "NA"
70      else:
71          try:
72              witness = (jason["witness"])
73          except (OSError, TypeError):
74              witness = "ERROR"
75      
76      if (jason["witness_signature"]) == []:
77          witness_signature = "NA"
78      else:
79          try:
80              witness_signature = (jason["witness_signature"])
81          except (OSError, TypeError):
82              witness_signature = "ERROR"
83      
84      if (jason["transactions"]) == []:
85          transactions = "NA"
86          operation_type = "NA"
87          transaction_ids = "NA"
88          print('\nI am processing block_id: ' + block_id)
89          ###PRINT COUNTER
90          print('COUNTER ' + str(counter))
91          ###EXECUTE FIRST LOOP SQL HERE
92          sql1 = ("INSERT INTO `data01`(counter, block_id, extensions, previous, signing_key, timestamp, transaction_ids, transaction_merkle_root, witness, witness_signature, operation_type, transactions, blockdata)VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)")
93          data1 = (int(counter), block_id, str(extensions), previous, signing_key, timestamp, str(transaction_ids), transaction_merkle_root, witness, witness_signature, operation_type, transactions, blockdata)
94          # Execute the SQL command
95          cursor.execute(sql1, (data1))
96          db.commit()
97          counter = int(counter) + 1

Do Not Be Intimidated By The Many Lines Of Codes

No worries here, as most of the lines are recurring if statements just to make sure that whatever data there can be captured without any errors that will stall our python script.

Before anything else let us be reminded that our main purpose is to save the steemit blockchain records into a known database (MySQL or MariaDB). Doing so with so many lines of data and going back and forth between the file and the database will be very tedious and time consuming for the computer.

Thanks to python, we can do these things fast and easy. We will discuss the routines line by line on the next post as this gets long already.

Stay glued, I know this gets exciting.