Dockerizing the Steem FOSSbot

If you are reading this, then you probably followed the tutorial posted by @codydeeds on setting up the Steem FOSSbot voter bot.

You may even have set it to run every 5 minutes like I did.

Unfortunately this leads to the exhaustion of the free dyno hours offered on the free tier of of the Heroku plans.

Now, Heroku does not show you how many free dyno hours you have left, so what are you to do when they run out?

I decided to try and see if I could set the voter bit up on a separate server. That would mean I did not need to worry about its resource usage nearly as much.
Over and above that I have an instance on Digital Ocean with spare capacity, so running the bot on there would cost me nothing extra anyway.

In this post I am making many assumptions as to knowledge levels and experience, as a post covering everything would be very lengthy indeed!

Assumptions are

• You have a Heroku instance of the voter bot
• You know what Git is
• You know what Docker is
• You have basic (at least) knowledge of a *nix OS and the command line
• You have the patience to sit through my ramblings ;)

Why Docker?

Docker container logo: By dotCloud, Inc. (File:Docker (container engine) logo.png) [Apache License 2.0 (http://www.apache.org/licenses/LICENSE-2.0)], via Wikimedia Commons
Voter bot logo courtesy of @personz, used with permission.

I work on a Macbook, but my server instance runs Centos. In theory, Docker allows me to set it all up and get it running locally with a tight, iterative cycle, then deploy that with ease on another machine running Docker.

Was it that easy? Read on and see ;)

The Docker way

Traditionally, servers have a multitude of services running on them, from web servers, databases caches and message brokers etc.

The Docker way is to have a container per 'service' that is running, let's look at the voter bot as an example.
The voter bot makes use of Node.js for the 'code' that drives the web servers as well as the bot that does the actual posting, additionally it makes use of Redis as a data store. Redis holds configuration as well as data on previous bot runs and votes cast etc.

If you were setting this up normally, you would just install Node.js along with Redis and be done, but because I decided to use Docker, I chose to have two separate containers, one for the Node.js portion and another for the Redis portion.

The steps

There are a few steps to getting the voter bot running in a docker container, locally or on a server.

1. Make sure Docker is installed and running
2. Create the files / directory structure below
3. Checkout the git repository of the Steem FOSSbot
4. Add the missing files to that directory
5. Run the docker-compose commands
6. Export / Import configs and algorithms
7. Have a cuppa tea!

1. Make sure Docker is installed and running

There is much written on the web around getting Docker installed, so I am just going to point you to the page where you can get started https://www.docker.com/products/docker

*For nix systems, please note you need to install docker-compose as well!
This is included in the Docker toolbox, which is installed on Mac's / Windows machines.

2. Create the files / directory structure below

For this to work out the box, you need the following directories and files;

• You root/working directory can be where ever, but needs to contain the docker-compose.yml file
• Inside your working directory you will have 2 directories, named redis and steem-fossbot-voter
• You will need to create the redis directory manually, bit the other one will be created in the next step

Go ahead and create docker-compose.yml with the following contents.

node:
    build: ./steem-fossbot-voter
    links:
        - redis
    ports:
        - "5000:5000"
redis:
    image: redis
    ports:
        - "6379:6379"

Then create the redis directory.
Inside the redis directory, create a file called Dockerfile (NO extension!)
Save the following bit of text into your newly created Dockerfile;

# Set the base image to Ubuntu
FROM        ubuntu:trusty

# File Author / Maintainer
MAINTAINER Shaun Morrow

# Update the repository and install Redis Server
RUN         apt-get update && apt-get install -y redis-server

# Run Redis Server
ENTRYPOINT  ["/usr/bin/redis-server"]

3. Checkout the git repository of the Steem FOSSbot

I have branched off of the original repository, but you don't need to do that.

Change back to your working directory, the one with docker-compose.yml in it.
If you have an account on Github.com, run the following command;

git clone [email protected]:Steem-FOSSbot/steem-fossbot-voter.git

If you want to check the code out anonymously, run;

git clone https://github.com/Steem-FOSSbot/steem-fossbot-voter.git

You should now have the directory called steem-fossbot-voter in you working directory, with all the CODEZ!!

Generated at imgflip.com

4. Add the missing files to that directory

Now change into the steem-fossbot-voter directory and create another Dockerfile file, this time save it with the following content;

# Set the base image to Ubuntu
FROM    ubuntu:trusty

# File Author / Maintainer
MAINTAINER Shaun Morrow

# Install Node.js and other dependencies
RUN apt-get update && \
    apt-get -y install curl && \
    curl -sL https://deb.nodesource.com/setup_4.x | sudo bash - && \
    apt-get -y install python build-essential nodejs

# Install nodemon
RUN npm install -g nodemon

# Provides cached layer for node_modules
ADD package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir -p /src && cp -a /tmp/node_modules /src/

# Define working directory
WORKDIR /src
ADD . /src

ADD crontab /etc/cron.d/bot-cron


RUN touch /var/log/cron.log
RUN chmod +x /etc/cron.d/bot-cron
RUN chmod +x /src/bot.sh

RUN /usr/bin/crontab /etc/cron.d/bot-cron


# Run app using nodemon
CMD /bin/sh /src/bot.sh

ENV COOKIE_SECRET "MAKE THIS A LONG, RANDOM STRING"
ENV BOT_API_KEY "YOUR KEY TO LOGON TO THE BOT"
ENV STEEM_USER "YOUR STEEM USER NAME"
ENV POSTING_KEY_PRV "YOUR PRIVATE POSTING KEY"
ENV REDIS_URL="redis://redis:6379"

You need to change this file with your own details or it will break.
I simply copied the values from my Heroku instance.

Be sure to update the following with your details;

ENV COOKIE_SECRET "MAKE THIS A LONG, RANDOM STRING"
ENV BOT_API_KEY "YOUR KEY TO LOGON TO THE BOT"
ENV STEEM_USER "YOUR STEEM USER NAME"
ENV POSTING_KEY_PRV "YOUR PRIVATE POSTING KEY"

---

Almost done with this part, two more files needed.

crontab (NO extension!), with the following contents;

0 * * * * root node /src/bot.js >> /var/log/cron.log 2>&1

This file determines how often the bot will run. As it is, the bot will run hourly on the hour. Change this as desired.

bot.sh with the following contents

 #!/usr/bin/env bash
 nohup nodemon /src/server.js > /dev/null 2>&1 &

  printenv | cat - /etc/cron.d/bot-cron > ~/bot-cron.tmp \
    && mv ~/bot-cron.tmp /etc/cron.d/bot-cron

echo "Running cron now ...."

cron -f

You should now have a directory structure similar to the following, highlighted are files that you would have created;

5. Run the docker-compose commands

Now you are ready to run the commands to get it up and running. Very excite! YESSS!

Generated at imgflip.com

First change back to your working directory, the one with the docker-compose.yml file in it.

When starting from a clean slate, we need to 'build' our containers first; so we start off with the following command;

docker-compose build

This will take quite a while the first time round, as it needs to fetch and build all of the images our containers are based off of. Depending on your internet connection you may want to give it quite a while!

You should see something similar when it completes;

All built? Now run;

docker-compose up

This tells docker-compose to start up all the containers based on the info inside of the docker-compose.yml file.

That should give something similar to the following output, you can press CTRL^C to exit that.

Finally, to run this as a background process, run the following command;

nohup docker-compose up > /dev/null 2>&1 &

You need to detach this process, or the containers will stop running when you close your terminal session.

Depending on where you are running this, you can navigate to the address in your browser, to see the voter interface.

For local installs the following should work;
http://127.0.0.1:5000

6. Export / Import configs and algorithms

With your containers now running you can export your configuration and algorithm from Heroku, then import it into your dockerized voter bot!

If you did not have a bot set up on Heroku, then you can skip this step and set up your config and algorithm as you choose.

To get started, first on Heroku, enter your BOT_API_KEY that you into the text field and click the Start Session button, seen below;

Now we can export the existing algorithm, click on the Edit >> button under the Edit Algorithm heading.

Scroll down till you see the Export Current Algorithm button , then click on it. It will populate the text area with the JSON representation of your algorithm. See an example below;

Now follow the same steps on your dockerized version, but paste the text into the text area. Then click on the Import This Algorithm button to persist the changes.
The page will refresh, and all the relevant fields will be populated.

For your configuration changes, you follow a very similar process, after clicking on the Edit >> button under the Edit Config heading. Copy your config from the existing Heroku instance to your new install.

7. Have a cuppa tea!

Looks like my cup is almost empty!

Now just sit back and relax, your bot will fire off soon, check your stats to see what it cast your votes on.

Conclusion

This implementation is by no means production ready, so use at your own risk.

I have a few changes I would like to make, but work / life / short attention span get's in the way.

Some improvements could be;

• Supervisord implementation in the containers. Right now the processes are started, if the fail, they fail and don't restart.
• Ensure containers are up and running.
• Link code rather than copying it in. ADD . /src in the steem-foss-voter/Dockerfile, with copying it in, ANY code changes require you to docker-compose stop then docker-compose build and docker-compose up. I think it would be much neater to just pull the code changes down, then restart the node server.js command somehow.

But what do I know? ;)

Credits;

• http://anandmanisankar.com/ I relied heavily on this post for the dockerfile's and docker-compose.yml file contents.
• Run a cron job with Docker - Cron job's in Docker, who would have thought it would be so hard!?!
• CMD vs. ENTRYPOINT - Dunno actually, I must have bookmarked it for a reason...
• A big thank you to @personz, for 1. making, then open sourcing an awesome bot. 2. All the help and support as I fumbled my way through Docker / Node.js and Redis, none of which I have any working experience with.

Whoot! You made it!
This is a long post, but I tried to make it a bit fun to read, I hope you got something out of it!

One more favour to ask, please be so kind as to drop a comment with your thoughts/criticisms, thank you.