Building a Blockchain with Go - Part 7 - The UTXO Set and BadgerDB Iterators

Repository

https://github.com/golang/go

What Will I Learn?

• You will learn about the UTXO set
• You will learn how to iterate over BadgerDB keys
• You will learn how to add perfixes to BagerDB
• You will learn about Blockchain optimization
• You will learn how to add a chainstate persistence layer
• You will learn how to batch delete keys from BadgerDB

Requirements

System Requirements:

Operating System:

• FreeBSD 10.3 or later
• Linux 2.6.23 or later with glibc
• macOS 10.10 or later
• Windows 7, Server 2008R2 or later

Required Knowledge

• A little understanding of the Go programming language
• Go installed on your computer
• A text editor or IDE like Gogland (VS Code used)

Resources for Go and this Project:

• Awesome Go Github: https://github.com/avelino/awesome-go
• Golang Installation Page: https://golang.org/dl/
• Golang Home Page: https://golang.org/
• Golang Documentation Page: https://golang.org/doc/
• Documentation about the Big library: https://golang.org/pkg/math/big/
• Documentation about the Bytes library: https://golang.org/pkg/bytes/

• Information about Elliptic Curves: http://mathworld.wolfram.com/EllipticCurve.html
• Information about Base58: https://en.bitcoinwiki.org/wiki/Base58
• BadgerDB Documentation: https://godoc.org/github.com/dgraph-io/badger
• MermaidJS: https://mermaidjs.github.io/

Credits/Sources:

• Go Logo: https://golang.org/

Difficulty

• Advanced

Description

Blockchains are by definition immutable and distributed databases. As we move closer and closer towards a production level application, we need to consider how we can avoid some of the pitfalls that could potentially happen from a lack of scaling. While our testing environment has only really dealt with four to five blocks at once; each with their own transaction sets, a real blockchain can scale out to hundreds of thousands of blocks which takes up a significant portion of disk space. Before we add the peer to peer networking layer to our application, we need to ensure that each peer doesn't have to hold and read 100-200 GBs worth of data just to verify a transaction on the blockchain.

UTXO Persistance

As we've seen in these tutorials, most of the transactions in a blockchain rely on what are called Unspent Transaction Outputs. If you want to know the balance of an account, you count up the tokens inside of the set of unspent outputs which are locked by the user's public hash key. If you want to send tokens from one account to another, you also need to use this set of unspent outputs to create that transaction. The unspent transaction output set is important and it makes sense to separate this layer of data from the blockchain itself. Not all users want to run a full node to access a blockchain. Even full nodes should not have to read the entire blockchain to make a new transaction or find an account's balance. This becomes even more important as the blockchain scales out to hundreds of thousands of blocks.

The flowchart above shows a three block blockchain. In this example, there are three unspent transaction outputs and eight total transactions in all three blocks combined. The blockchain persistence layer stores all of the transactions and each block that contains those transactions. The UTXO set on the other hand only stores the Unspent Transaction Outputs to minimize the cost of having to create new transactions. In the case of Disk space and Performance, using the UTXO for most transactions is better and is no less secure then using the blockchain itself. This persistence layer is called the chainstate persistence layer and it is one of the major ways of optimizing and shrinking the size of a blockchain node.

Differentiating Data in BadgerDB

This UTXO set needs to be stored in the database along with our blockchain. With BadgerDB being a low level Key-Value storage solution, the main way of doing this is by using prefixes. Badger does not contain any tables nor does it have buckets but we are able to append prefixes to the keys of like data to properly work with and modify that data in bulk. It might seem intuitive to create a separate database for the chainstate apart from the blockchain database, but this would require adding a second instance of the BadgerDB database client to the application. Doing this is possible but it removes any guarantees that the application will not corrupt the data in the database.

In our code, we built a DeleteByPrefix helper function. The primary function of this method is to iterate through our database and delete keys that are associated with a specific prefix by bulk. The function which is featured in the code above, opens a read-only transaction with the BadgerDB database. It then iterates through the database without fetching the values associated with the database keys. Once its collected up to one hundred thousand keys, it executes a bulk delete command to remove both the keys and values from the database. If there are more than one hundred thousand keys in the database, then the command deletes the keys in iterations of one hundred thousand at a time. This is a much more efficient process then deleting each key individually as they are being read. This function is used to reindex the UTXO set in our database as it is created and as it grows.

The Source Code for this video may be found here: https://github.com/tensor-programming/golang-blockchain/tree/part_7

Video Tutorial

Curriculum

• Building a Blockchain with Go - Go Modules and a Basic Blockchain - Part 1
• Building a Blockchain with Go - Refactor and Proof of Work - Part 2
• Building a Blockchain with Go - Persistence and Command Line - Part 3
• Building a Blockchain with Go - Adding Primitive Transactions - Part 4
• Building a Blockchain with Go - Part 5 - Building a Basic Wallet Module
• Building a Blockchain with Go - Part 6 - Adding Digital Signatures

Proof of Work Done

https://github.com/tensor-programming