A continuation of my Bash Script Examples series, continuing on with what I hope are some fun examples.

This tutorial will revisit some topics from Bash Script Examples (very very basic) and provide some more useful examples. Originally I was going to go in depth with here docs and here strings as well, but I quickly hit the 1K word limit and decided to split that into a separate post and then just add more useful examples here. Or, if not useful, at least examples that can be run from the shell to interact with steemit.com in a fun way. The result is a post of lexian length, and with any luck, value.

Topics Covered

Redirection Via:

• Pipes
• Files
• Processes

I'll use some strange jargon, hopefully this table renders it less strange.

Feel free to skip around in the article, but be warned, successive examples generally build from the initial ones.

Redirection

With a Pipe

You can send the output of a command into a pipe and it will be read by the command named after the pipe as though it were reading standard in.

wget https://steemit.com/@not-a-bird.json -O - | zcat

Redirection via a pipe.

Here the JSON document describing my account is fetched using wget, when wget is passed a - for the -O option it writes the fetched file to standard out. The fetched file is actually gzipped. So to make it readable, the output of the fetch is being sent through the zcat command where it will be unzipped. The result is displayed on the screen.

{"user":{"id":410715,"name":"not-a-bird","owner":{"weight_threshold":1,"account_auths":[],"key_auths":[["STM6fDBdfZdLocbnRMhDXBJJN8FdsJhckeKEnNpojmp6fDd512sse",1]]},"active":{"weight_threshold":1,"account_auths":[],"key_auths":[["STM6Kk3cGQm9NV41npvZzjtfivk1SMjZHHmzuL7u4NzhZaTSvE3E3",1]]},"posting":{"weight_threshold":1,"account_auths":[["busy.app",1],["steemauto",1],["utopian.app",1]],"key_auths":[["STM8CPj4Gyv6cbG6YWuFCzaN68G8FuPU9UePLoxkWAB2SkRyWmxkP",1]]},"memo_key":"STM7ijo1JJx4pPiduYsz1tmoSqx8ghpsxeYBVJpLUVFbKMQQN6Drw","json_metadata":{"profile":{"profile_image":"https://steemitimages.com/DQmPLUfGWJue6DKbUfE96f1fGF3pasHaQX1zJBW3Xqe5NsT/av2.png","cover_image":"https://i.imgur.com/RVk3hwx.png"}},"proxy":"","last_owner_update":"1970-01-01T00:00:00","last_account_update":"2017-12-31T21:59:27","created":"2017-10-17T22:35:06","mined":false,"owner_challenged":false,"active_challenged":false,"last_owner_proved":"1970-01-01T00:00:00","last_active_proved":"1970-01-01T00:00:00","recovery_account":"steem","last_account_recovery":"1970-01-01T00:00:00","reset_account":"null","comment_count":0,"lifetime_vote_count":0,"post_count":493,"can_vote":true,"voting_power":8573,"last_vote_time":"2018-01-14T23:37:18","balance":"5.897 STEEM","savings_balance":"1.000 STEEM","sbd_balance":"2.920 SBD","sbd_seconds":"3057221478","sbd_seconds_last_update":"2018-01-14T23:17:54","sbd_last_interest_payment":"2017-12-28T00:04:30","savings_sbd_balance":"0.000 SBD","savings_sbd_seconds":"0","savings_sbd_seconds_last_update":"1970-01-01T00:00:00","savings_sbd_last_interest_payment":"1970-01-01T00:00:00","savings_withdraw_requests":0,"reward_sbd_balance":"0.000 SBD","reward_steem_balance":"0.000 STEEM","reward_vesting_balance":"6.143331 VESTS","reward_vesting_steem":"0.003 STEEM","vesting_shares":"524922.261450 VESTS","delegated_vesting_shares":"325881.236324 VESTS","received_vesting_shares":"222015.000000 VESTS","vesting_withdraw_rate":"0.000000 VESTS","next_vesting_withdrawal":"1969-12-31T23:59:59","withdrawn":0,"to_withdraw":0,"withdraw_routes":0,"curation_rewards":1881,"posting_rewards":207329,"proxied_vsf_votes":[0,0,0,0],"witnesses_voted_for":5,"average_bandwidth":"142786125958","lifetime_bandwidth":"1324125000000","last_bandwidth_update":"2018-01-14T23:37:18","average_market_bandwidth":"5369922975","lifetime_market_bandwidth":"68400000000","last_market_bandwidth_update":"2018-01-12T14:31:27","last_post":"2018-01-14T06:17:06","last_root_post":"2018-01-14T02:20:00","vesting_balance":"0.000 STEEM","reputation":"1969255158997","transfer_history":[],"market_history":[],"post_history":[],"vote_history":[],"other_history":[],"witness_votes":["furion","jesta","klye","ocd-witness","utopian-io"],"tags_usage":[],"guest_bloggers":[]},"status":"200"}

Sample output from the previous command.

Or, if we want it in a prettier format, we can pipe the result of that through either json_pp or jq.

wget https://steemit.com/@not-a-bird.json -O - | zcat | jq

Example of redirection through multiple pipes.

{
  "user": {
    "id": 410715,
    "name": "not-a-bird",
    "owner": {
      "weight_threshold": 1,
      "account_auths": [],
      "key_auths": [
        [
          "STM6fDBdfZdLocbnRMhDXBJJN8FdsJhckeKEnNpojmp6fDd512sse",
          1
        ]
      ]
    },
    "active": {
      "weight_threshold": 1,
      "account_auths": [],
      "key_auths": [
        [
          "STM6Kk3cGQm9NV41npvZzjtfivk1SMjZHHmzuL7u4NzhZaTSvE3E3",
          1
        ]
      ]
    },
...

Example output from wget https://steemit.com/@not-a-bird.json -O - | zcat | jq

An additional useful bit of syntax is the |& which will redirect standard error and standard out into whatever command follows. I wont be providing an example, but I mention it in case you see it in the wild.

You may have guessed, but you can substitute your own Steem name in place of not-a-bird.

With a File

Redirecting Output

You can redirect command output to a file. From the Bash manual, the general form is

[n]>word

This means you can write from file descriptor n and it will write over the file named by word. Generally you wont provide an n and you will just be sending standard out to the file.

Here's an example, the builds on the previous:

wget https://steemit.com/@not-a-bird.json -O - | zcat > user.json

The wget command fetches gzipped JSON, it's passed to zcat to unzip it, then the output of zcat is sent to the file named user.json. If you view user.json in a text editor, you will see the same JSON blob as was produced by the Redirection → With a Pipe example.

In addition to providing a file name as the word value, you can also also provide a file descriptor using >&[n], for example, in an earlier tutorial I redirected standard out to standard error this way:

echo "hello" 1>&2

Example from Bash Script Examples (very very basic).

This outputs "hello" but sends it to standard error.

Redirecting Input

You can redirect the contents of a file to standard input and run that file content through the command as though you were typing that file's content into the command.

From the Bash manual, the general form is

[n]<word

This means you can read from the file named by word and any access to file descriptor n will instead read from the specified file. Generally you wont provide an n and instead you'll just be sending standard in to the file.

jq '.user.balance' < user.json

Example of input redirection from a file.

You will need to have done the example in With a File → Redirecting Output for this example
to work. Also, a more complex example that provides a value for n can be found in Processing Lists of Files.

Here, the jq command will try to read a JSON formatted string from standard in and look for the balance field under the user object. Because of the redirection from user.json it will get the contents of user.json when it reads standard in.

Example output:

{
  "user": {
    "id": 410715,
    "name": "not-a-bird",
    "owner": {
      "weight_threshold": 1,
      "account_auths": [],
      "key_auths": [
        [
          "STM6fDBdfZdLocbnRMhDXBJJN8FdsJhckeKEnNpojmp6fDd512sse",
          1
        ]
      ]
    },
    "active": {
      "weight_threshold": 1,
      "account_auths": [],
      "key_auths": [
        [
          "STM6Kk3cGQm9NV41npvZzjtfivk1SMjZHHmzuL7u4NzhZaTSvE3E3",
          1
        ]
      ]
    },
...

Process Substitution

Redirecting Output

Very similar to file redirection, you can redirect the output of a process to a file descriptor and use it in lieu of a file name to some command as output.

From the Bash manual, the general form is

>(command)

So, to accomplish one of the above commands of fetching user JSON and unzipping
it with zcat:

wget https://steemit.com/@not-a-bird.json -O >(zcat)

Example of output redirection to a process.

You should see the same JSON blob as was produced by the Redirection → With a Pipe example.

Redirecting Input

Very similar to file redirection, you can redirect the output of a process to a file descriptor and use it in lieu of a file name to some command as input.

From the Bash manual, the general form is

<(command)

So, to accomplish one of the above commands of fetching user JSON and unzipping it with zcat:

zcat <(wget https://steemit.com/@not-a-bird.json -O -)

Example of input redirection from a process.

You should see the same JSON blob as was produced by the Redirection → With a Pipe example.

Fun Examples

Cryptocompare.com

Did you know you can fetch any given coin and get it's price in any other coin or fiat?

wget 'https://min-api.cryptocompare.com/data/price?fsym=STEEM&tsyms=USD' -O -

Get the price of STEEM in USD.

If you run this command, the output is quite messy. So messy, in fact, that
standard error will most certainly make it impossible to see standard out. So
we will use file based redirection to throw it away.

wget 'https://min-api.cryptocompare.com/data/price?fsym=STEEM&tsyms=USD' -O - 2>/dev/null

Get the price of STEEM in USD, but throw away standard error.

Here we have redirected standard error via file descriptor two and sent it to the special file /dev/null so that we wont have to see it.

But that's still got some JSON around it, so let's strip that off using standard tools that exist on just about every *Nix system.

wget 'https://min-api.cryptocompare.com/data/price?fsym=STEEM&tsyms=USD' -O - 2>/dev/null | cut -f2 -d:| cut -f1 -d}

Fetching the price of STEEM in USD, and throwing away the JSON wrapper with cut.

If you have jq on your system, you could use that more simply to remove the wrapping.

wget 'https://min-api.cryptocompare.com/data/price?fsym=STEEM&tsyms=USD' -O - 2>/dev/null | jq '.USD'

Fetching the price of STEEM in USD, and throwing away the JSON wrapper with jq.

Let's get the price of SBD while we're at it.

wget 'https://min-api.cryptocompare.com/data/price?fsym=SBD&tsyms=USD' -O - 2>/dev/null

Steemd.com

Sure, there are some nifty JSON RPC services that could be leveraged, but as an example for scraping web pages, let's say we wanted to know the value of STEEM per MVests?

Well, there's a handy place for that on steemd.com. And if we fetch the home page, we can use grep (the GNU Regular Expression tool) to pull that data out.

wget "https://steemd.com" -O - 2>/dev/null | grep -o '<samp>steem_per_mvests</samp></th></tr><tr><td><i>[0-9]\+\.[0-9]\+</i>' | grep -o '[0-9]\+\.[0-9]\+'

Fetch steem_per_mvests from steemd.com.

Let's disect this. The wget command will pull down the steemd.com index page, sending standard error to /dev/null and standard out on to the first grep command. The first grep will locate the bit of HTML that contains the steem_per_mvests table heading and the actual figure. The -o option causes it to only return the content that matched the expression, so it will discard the rest of the page before sending that result to the second grep command. The second grep command will locate any digits separated by a decimal point, again, using the -o option, so it will carve out just the matching number.

488.341

Example output of the wget off steemd, piped through greps to extract the steem_per_mvests value.

Steemit.com

In previous examples I fetched a user's profile, but other than displaying it, I didn't do much with it. I also fetched the steem_per_mvests. Now let's do something fun with this. Let's get a more realistic value of what your account is worth than even what steemit.com provides.

We'll use the coin values for SBD and STEEM from cryptocompare and then we'll do some match with the steem_per_mvests and we'll add up and multiply and have a USD dollar amount that's fairly accurate.

First, we need to get the coin totals from your wallet. We can do this by fetching the user json, unzipping it, and looking at the following fields, they're fairly self explanatory, but I'll be explicit anyway because their names are all so similar.

VAR=$(command)

So, first, get the user JSON and store it in a file, uncompressed, but let's not be sloppy, we'll do this so we can build a script. First, we need a place to store the file.

WHERE=$(mktemp)

Okay, we can use "${WHERE}" as the file name. That way we can parse each field we care about one at a time without having to refetch the value.

wget https://steemit.com/@not-a-bird.json -O - 2>/dev/null | zcat > "${WHERE}"

Now we need to use jq to pull out the values we care about, but some of them will have string suffix of "STEEM" or "SBD" so that will need to be stripped off. We'll use cut for that. So, let's get the balance.

BALANCE=$(jq '.user.balance' < "${WHERE}" | cut -f2 -d'"' |  cut -f1 -d" ")

Now we have the STEEM count for the user.

Now we need the sbd_balance, again, extracted with jq and stripped with cut.

SBD_BALANCE=$(jq '.user.sbd_balance' < "${WHERE}" | cut -f2 -d'"'| cut -f1 -d" ")

Now we have the SBD count for the user.

Now we need steem_savings, again, extracted with jq and stripped with cut.

STEEM_SAVINGS=$(jq '.user.savings_balance' < "${WHERE}" | cut -f2 -d'"'| cut -f1 -d" ")

Now we have the STEEM savings count for the user.

Now we need the vesting_shares so we can calculate the SP.

VESTING_SHARES=$(jq '.user.vesting_shares' < "${WHERE}" | cut -f2 -d'"'| cut -f1 -d" ")

Now we have vests for computing SP.

Okay, but that's in terms of vests, and we need MVESTS, and we need to multiply that by the steem_per_mvests from steemd.com. First, getting that steem_per_mvests, just like before:

STEEM_PER_MVESTS=$(wget "https://steemd.com" -O - 2>/dev/null | grep -o '<samp>steem_per_mvests</samp></th></tr><tr><td><i>[0-9]\+\.[0-9]\+</i>' | grep -o '[0-9]\+\.[0-9]\+)'

Now we have the second number needed for computing SP.

Now, let's do some math with bc to calculate SP:

STEEM_POWER=$(echo "${VESTING_SHARES}*${STEEM_PER_MVESTS}/1000000.0" | bc -l)

Now we have the SP.

This multiplies the VESTING_SHARES by the STEEM_PER_MVESTS, but that has to be divided by a million because the balance was in terms of vesting shares, not millions of vesting shares. Additionally, we must use bc -l so that it loads math libraries. This shouldn't have been necessary, or at least, that's what the man led me to believe, but in testing it was necessary. We used the echo command to build up the math problem, and then redirected this to the bc command to have it give us the number, the result was stored in STEEM_POWER.

Next, let's get the value of STEEM in USD.

STEEMV=$(wget 'https://min-api.cryptocompare.com/data/price?fsym=STEEM&tsyms=USD' -O - 2>/dev/null | jq '.USD')

Now we have the value of STEEM in USD.

And the value of SBD in USDr.

SBDV=$(wget 'https://min-api.cryptocompare.com/data/price?fsym=SBD&tsyms=USD' -O - 2>/dev/null | jq '.USD')

Now we have the value of SBD in USD.

Now we have all these variables, we need to add up the STEEM, SP, and SP in savings, then multiply that by the value of STEEM in USD. Then take that and add it to the SBD (times the value of SBD) to get the total account value.

BANK=$(echo "(${BALANCE}+${STEEM_POWER}+${STEEM_SAVINGS}) * ${STEEMV} + ${SBD_BALANCE} * ${SBDV}" | bc -l)

Now we have the total account value in USD.

The total account worth can now be retrieved from the ${BANK} variable.

echo "${BANK}"

Displays the total account value in USD.

Remember that temporary file we created with the user's profile data in JSON? We can delete that now that we're done with it.

rm "${WHERE}"

Deletes the temporary file.

Putting it all together... I created a project on github where I'll be compiling all the Bash scripts and functions that I create for steem from now on. You can get this one here.

Conclusion

We covered the following topics:

• Pipes
• Files
• Processes

If you followed through with all of the examples, you're either incredibly confused or you have an excellent grasp of all the concepts covered here!

References:

• Bash manual
• BC manual
• wget manual
• grep manual
• cut manual
• zcat manual
• jq user manual