This is the fourth part of a 9 part tutorial series about the R Statistical Programming Language, targeted at data analysts and programmers that are active on Steem.

In this course you will learn the R programming language through practical examples.

Repository

The R source code can be found via one of the official mirrors at

• https://cran.r-project.org

This tutorial is part of a series, the text as well as the code is available in my github repo

• https://github.com/kharoof/R-Tutorials-for-Steemit

What Will I Learn?

The first two tutorials introduced the basics of R and the free R Studio IDE, the rest of this series will focus on worked examples where we will slowly introduce new concepts and learn some of the extended functionality of the R programming language.

In this tutorial you will:

• Get an intro to Data Wrangling with R
• Learn techniques to Explore Raw Data with R
• Learn about Regular Expressions in R
• Learn how to Manipulate Dates in R
• Extract Clean and Visualise a Raw Dataset

Requirements

• Lesson 1 Introduction to R
• Lesson 2 Finding Your Way Around R
• Basic Knowledge of Statistics
• Basic Programming Experience
• Some Knowledge of Steem and Cryptocurrencies

Difficulty

Intermediate

Data Wrangling

80% or more of a data analysis project can be spent on this first stage, data cleaning.

What is Data Wrangling?

Data that has been collected is usually in a raw unprocessed messy format. The first stage of data analysis involves detecting and correcting corrupt or inaccurate data from a dataset and standardising the formatting and meaning of the data.

• Data can be corrupt if it is incomplete, inaccurate or irrelevant.
• Data may also be in the incorrect format for analysis.

How does R help?

• R enables you explore your data using statistical summary tools and visualizations.

• R excels in data manipulation, it has general flexibility common to other languages including tools such as Regex, but packages extend generic functionality and provide a concise syntax (low cognitive) for data cleaning and manipulation. The most advanced set of packages are known as the TidyVerse set of packages where data processing actions are expressed as verbs.

The TidyVerse ...

Apart from mentioning it, I will not talk much about the TidyVerse ecosystem of opinionated packages much in this tutorial. For our introduction to data cleaning with R we we will focus on using features of the data.table package and get an introduction to regular expressions and a useful date package for cleaning our data.

Data.Table

The data.table package is a good place to start in R for data wrangling for two reasons:

• It has a familiar structure, providing a table or database like structure to the data. It is interchangeable with a data frame format which is a commonly used data storage framework in R.
• It is extremely fast.
• It works with efficiently with very large datasets.

Exploring the Data

Before modifying of fixing any of our data we can use some functionality from R to have a look at it. We will work through some of these tools using live data from the Steem Blockchain.

***To begin we will get some interesting data using the steemRdata package. ***

We will first load our familiar R Studio setup. If you need to refresh about R Studio, you may look back to Lesson 2 for a refresher.

Open R Studio and open a new untilted script where we will type our code

Load the steemRdata package by typing the following in the script and hitting ctrl + Enter to send the command to the console

library(steemRdata)

For this exercise we are going to look at delegations made by @blocktrades and see what insights we can get by exploring the data.

We will now get all the delegations that are being made by the @blocktrades account
Type the following in our script and send to the console

blocktrades <- getDelegation("blocktrades")

• This will retrieve data about all the delegations from the Blocktrades account and store it in a variable "blocktrades".

• You will see a variable appear in the upper right area of R Studio

Exploring the Data

Typing the variable name in the console gives us a summary of the data.

blocktrades

• From this we can begin to understand the data, we can see how many rows, some data items etc.

To take this a little further we can get more details of the fields using the str (structure) command

str(blocktrades)

• We can now see that some of the fields have numeric values and some are characters.

You could also see the same information by expanding the blue button in the Environment window

We now have our data loaded from the blockchain

Next we can get a statistical overview by typing the summary command and sending it to the console

summary(blocktrades)

• For each field we get a statistical summary
• We get a 5 number summary if the data fields is a numeric field
• If the field is a character we get less info, here just the the size of the field.
• If the field is a factor (which we will discuss in later tutorials) we get a cross tab summary.

Today I am going to look at how much SP @blocktrades is delegating to different accounts. I can access the SP field with the following command

blocktrades$SP

• This shows all the SP for each delegatee in the console but using R we can build in visual statistical summaries.

To view a histogram of the data type

hist(blocktrades$SP)

• Here you can see the distribution of the SP, lots of small delegations and some very big ones.

To view a boxplot of the data

boxplot(blocktrades$SP)

• The boxplot is another way of representing the distribution of the SP, again lots of small delegations and some very big ones.

At this stage we have explored the data. The data is complete, there are no errors but there may be extra columns we would like to add, or subsets of the data we wish to look at.

Regular Expression

Regular expressions provide tools for searching and manipulating strings which are extremely useful.

We mentioned that R has many general purpose tools such as Regular Expressions but it extends the generic functionality with add on packages.
In R the stringr package makes working with regular expressions really easy. If this is not installed we can install it before loading it.

install.packages("stringr")

Load the package

library(stringr)

We will now check each delegatee to see if they have the word “steem” in the username using the "str_detect" function.

blocktrades[str_detect(delegatee, "steem")]

• str_detect is a function from the stringr package which takes two arguments. The field to search, and the regular expression to search for, in this case the word steem. The function will return true or false if it matches the string. This filters our result to just show the rows of the blocktrades variable that have the matching condition
• We found 23 delegatees with the word steem in their name

You can tidy this up into one line and chain multiple expressions to the data table syntax

blocktrades[str_detect(delegatee, "steem")][,"delegatee"]

This is very basic example of a regular expression but it is a really powerful tool that is seamlessly integrated into R. It is ideal for parsing blockchain data and building up complex regular expressions to extract text that you are looking for from json_metadata.

Working with Dates

Dates are fun in any programming language, but in R there are packages that make working with dates very intuitive. The package we are using today is the lubridate package.

install.packages("lubridate")
library(lubridate)

We are going to use this package to add a year and month column to our dataset so that we can summarise and plot monthly SP delegations.

Adding a column in a data.table we can use the := command which performs row wise operations on the data.table. In this case we apply the month and year function and store them in new fields "del_month" and "del_year"

blocktrades[, del_month:=month(date)]
blocktrades[, del_year:=year(date)]

Plotting the Data/Modelling

For illustration purposes today we are going to plot monthly totals for SP delegated in 2018.

At this stage in our workflow we have explored our data, cleaned it and and added a few useful columns which we are now going to use to plot the data.

To plot our data we will use the ggplot2 package from previous tutorials.

library(ggplot2)

We will next use the ggplot function to make a bar plot using the geom_col() function.

ggplot(blocktrades[year==2018], aes(x=month, y=SP))+geom_col()

• In this command we have filtered the data where del_year is 2018 i.e. blocktrades[del_year==2018]
• We specify the aesthetics, the x variable is the month, and the y variable is the Steem Power Delegated or SP i.e. aes(x=month, y=SP)
• We use the plotting function geom_col() which creates a bar chart based on our data

To see how many delegations were made each month you can use a very similar plotting command but with a different geometry

ggplot(blocktrades[del_year==2018], aes(x=del_month))+geom_bar()

• geom_bar() counts the number of observations in each bucket, in this case the buckets are specified by the month.

---

Recap

In this lesson:

• Get an intro to Data Wrangling with R
• Learn techniques to Explore Raw Data with R
• Learn about Regular Expressions in R
• Learn how to Manipulate Dates in R
• Extract Clean and Visualise a Raw Dataset

Benefits of Using R for Data cleaning

• Powerful Regular Expression Support,
• Support For Dates and Other Awkward Formats
• Fast rowwise Data.Table Operations on Large Datasets
• Advanced Best Practice Workflows Built into Package Design

Code Used

Get Data from the Steem Blockchain

library(steemRdata)
blocktrades <- getDelegation("blocktrades")
blocktrades

Explore the Data

str(blocktrades)
summary(blocktrades)
blocktrades$SP
hist(blocktrades$SP)
boxplot(blocktrades$SP)
install.packages("stringr")

Run regular expressions to filter the data

library(stringr)
blocktrades[str_detect(delegatee, "steem")]

Add features to the data

install.packages("lubridate")
library(lubridate)
blocktrades[, del_month:=month(date)]
blocktrades[, del_year:=year(date)]

Visualise the data

library(ggplot2)
ggplot(blocktrades[del_year==2018], aes(x=del_month, y=SP))+geom_col()
ggplot(blocktrades[del_year==2018], aes(x=del_month))+geom_bar()

Coming up

This course will cover the basics of R over a series of 9 lessons. We began with some essential techniques (in the first 2 lessons) and I will take you on a tour of some of the more advanced features of R with worked examples that have a Cryptocurrency and Steem flavour.

Rich data structures are one of the key features of R. In the next lesson we will explore time series data structures which will allow us to work efficiently and intuitively with time series data.

We might even make a few predictions about the price of Steem!

Curriculum

For a complete list of the lessons in this course you can find them on github. Feel free to reuse these tutorials but if you like what you see please don't forget to star me on github and upvote this post.

• https://github.com/kharoof/R-Tutorials-for-Steemit/

Related Posts

• Lesson 1 Introduction to R
• Lesson 2 Finding Your Way Around R
• Lesson 3 Scraping Web Data with R

---

Thank you for reading. I write on Steemit about Blockchain, Cryptocurrency and Travel.
R logo source: https://www.r-project.org/logo/