Witness Node Maintenance Series Part 1: Security on Ubuntu 16.04 Server

in #decent7 years ago (edited)

Don't blow off another's candle for it won't make yours shine brighter.

It has been almost a month now since I brought to life a Decent Witness node. After much discussion and motivation from @tjpezlo, we finally executed the first phase of The Plan - a plan to make a world a better place for you and for me and the entire human race (nah! pfft! we just want to make an app for the Decent Network).

This is the first part of the Maintenance Series which aims to guide readers and like minded individuals who want to setup their own Decent Witness Node. Part 1 of this series is not specific to the Witness node, rather this will guide you through some general maintenance tasks of running a server in general.

Server Status

To begin with, let us connect to our server through ssh ( shell access) and check storage spaces
by using the two commands:
$ ssh decent
$ df -H
The first command creates a shell access to the remote server, I have used an alias here which can be configured at ~.ssh/config, this is a shortcut to using <user@server-address>.
Screen Shot 2017-09-06 at 2.21.57 AM.png
Once logged in to the server df -H command shows the status of storage spaces on the server. As we can see, since I started running this node, the decent blockchain is taking up around ~22GB of my SSD. Now, I want to see how the server is taking up its load. To do this we issue the top command which shows the screen below.

Screen Shot 2017-09-06 at 2.23.13 AM.png

The load average is showing almost at ~0.00 which means the server is taking a stroll in the park. Only decentd is taking much of the server's processing power and memory. Although I am not yet an active witness (so please vote for me ), the role of the decent daemon (decentd) is to broadcast transactions, download blocks, and optionally sign them.

Witness Status

Now, let's check how my witness is doing. First, I need to run the cli_wallet which by default is connecting to the local node. The cli_wallet is the interface connecting us to the DECENT Network. It's responsible for creating accounts, broadcasting transactions, signing transactions, account retrieval etc. If you want to know further uses of the cli_wallet see here. As of this writing I gathered around 4.8 TV (teravotes) and very hopeful that the community will help me and vote for me to be an active witness.
Screen_Shot_2017-09-06_at_2_24_28_AM.png

Now unto the good stuff...

Basic Security Audit

This part of the Maintenance Series will briefly tackle the most basic but nonetheless important techniques/tips on securing your server.

  • Background processes and cron jobs

A common way of an attacker to maintain access to a compromised server is to hide and run a backdoor and add it as a cron job.

A cron job is a Linux utility used for scheduling a task to be executed in the specific time according to its schedule at designated time. (source)

To inspect your cron jobs use the command:
crontab –l - this will show the list of cron jobs running under the current user
crontab -l <username> - will show the list of cron jobs run by a specific user

To check the current running tasks/processes use this command:
ps ax - this will show you the list of processes, it's id and the user running it.

For both inspections, one thing to watch out for is an unrecognized user or process. An active backdoor normally hides on system directories and have access rights similar to the root (admin) user.

  • Network Connections

Below an audit of my server to check incoming and outgoing network connections. The mostly redacted column is my server's ip address and opposite to it is the external ip address that have established a TCP connection. The rightmost column is the process id and the process name responsible for the connection. Now we can see decentd is the only process creating these connections and we know that this is the decent daemon running which is as expected. I was also logged in through ssh hence, the connection on port 22. If you suspect an active backdoor process based from the previous inspections (cron and background process) it will show up in here.
cto_—_ssh_decent_—_99×39.png

  • SSH / auth logs

Now here comes the interesting part!
cto_—_ssh_decent_—_129×35.png
Brute force attack is one of the most common attack against servers. This is the endless attempt to guess the username/password combination to gain access to the server. To filter out the possible brute-force attack issue the following command on the terminal:
grep sshd.\*Failed /var/log/auth.log - this will search the auth.log file for "Failed" login attempts as shown in the photo above. Interesting enough, my server has been subject to these attacks. In the next posts I will show you how to harden a server to prevent these attacks.

For now, Good luck to these script kiddies, I could be running a honeypot somewhere right? Bring it on!

But just out of interest as Sun Tzu's Art of War, let us know our enemy.
With the help of my geoip server I was able to query some of these ip addresses:

  1. {"ip":"121.18.238.119",
    "country_code":"CN",
    "country_name":"China",
    "region_code":"13",
    "region_name":"Hebei",
    "city":"Hebei",
    "zip_code":"",
    "time_zone":"Asia/Shanghai"
    ,"latitude":39.8897,"longitude":115.275,
    "metro_code":0}

  2. {"ip":"109.201.152.8",
    "country_code":"NL",
    "country_name":"Netherlands",
    "region_code":"GE",
    "region_name":"Provincie Gelderland",
    "city":"Rozendaal",
    "zip_code":"6891",
    "time_zone":"Europe/Amsterdam",
    "latitude":52.0074,
    "longitude":5.9654,
    "metro_code":0}

  3. {"ip":"200.77.190.120",
    "country_code":"EC",
    "country_name":"Ecuador",
    "region_code":"P",
    "region_name":"Provincia de Pichincha",
    "city":"Quito",
    "zip_code":"",
    "time_zone":"America/Guayaquil",
    "latitude":-0.2167,
    "longitude":-78.5,
    "metro_code":0}

  4. {"ip":"59.45.175.11",
    "country_code":"CN",
    "country_name":"China",
    "region_code":"21",
    "region_name":"Liaoning",
    "city":"Shenyang",
    "zip_code":"",
    "time_zone":"Asia/Shanghai",
    "latitude":41.7922,
    "longitude":123.4328,
    "metro_code":0}

  5. {"ip":"190.48.139.176",
    "country_code":"AR",
    "country_name":"Argentina",
    "region_code":"C",
    "region_name":"Buenos Aires F.D.",
    "city":"Buenos Aires",
    "zip_code":"34034",
    "time_zone":"America/Argentina/Buenos_Aires",
    "latitude":-34.6033,
    "longitude":-58.3816,
    "metro_code":0}

I can see i'm quite popular and this is just the last few lines of the logs.

  • Other log files

  1. /var/log/syslog - shows system operations, root access, cron jobs, system info/debug logs
  2. /var/log/ufw.log - if you are using and have installed Uncomplicated Firewall, this log file shows the blocked traffic

Moving Forward

For the next part of this Maintenance Series, I will tackle some of the most complicated attack vectors which could compromise your servers. Mitigations and prevention techniques will also be discussed with case studies particularly focusing on my Decent Witness Node. I will also discuss the concept of Offensive Security.

In these posts I aim to share my experience and offer a guide to people who might find this useful in maintaining their own servers. If you have comments, suggestions and reactions just comment below and let me know your thoughts. If you think this post is helpful upvote or resteem so anyone who needs it, will find it.


Random Thoughts to Ponder:

email.png

I code with purpose. I am @precise.

Sort:  

Awesome post!

Detailed, very much so! And random thoughts so profound!

Profound thoughts on the blockchain.

if only...

Life full of what if's and if only's. Sigh.

No. I disagree.. Life is what you make it.

That is my motto.

I don't get it but it looks damn cool. Lol. Keep it up!

Thanks boss @deveerei ... i'm not a natural writer 😬 i should learn from your posts hehe. pardon my composition and i will try be more succinct next time. 😇

Not an issue. My mind's just fried haha I can't grasp info about Decent yet. But your post is great, superb in fact, explains a lot of details. It's just jargon to me but your posts are giving me glimpses into the Decent world.

how long did it take to get your node up and running, and what are the specs of your server. I have been trying to get steemd to catch up for the last 3-4 days, and i downloaded the entire block_log and ran steemd --replay.
I was under the impression that it would only take 12 hours to catch up, but 3-4 days later and its still under 30% :(

I got this up and running in less than an hour actually and I was able to do RPC calls to the node. By the way this is decent blockchain (decentd) although they're (steem and decent) both based on graphene. The specs of that server is -- 4-core processor, 120 GB SSD and 8GB RAM and it's a VPS. I have not encounter the same problem but if i'm in your situation I would delete the data file/directory and restart the steemd to sync again.

aha, that could be my issue then, I am trying to set mine up in a virtual machine (I sandbox everything) 4gb ram but only 1 core of my cpu, and my HDD is working at full tilt though this is likely down to having only 1 core.
Thank you for the tip, I shall try assigning the VM some more power and restart the process, hopefully that speeds things up dramatically. :)

you're welcome. hope it works out but virtualization will definitely cause you some problems. If you want to "sandbox everything", i'd recommend you use Docker instead. Let me know how it goes... have fun :)

Thank you, if you don't hear back from me then I must have gave up with it, but hopefully more power will sort it. :D

Don't give up. If you do, you will never know how close you were to finishing it.

Its looking like I'm going to have to put it on its own machine if I'm ever going to get it working. Another 3 days and still not at 30% and it now has 4 cores to work with :/

maybe it's a memory issue have you tried playing around with the swap file and the shared_memory? you can try to check free -h if you have enough to run the --replay I had it at 12GB but my RAM is just 8GB so i had to increase my swapfile to 6GB.

Congratulations! This post has been upvoted from the communal account, @minnowsupport, by dreamiely from the Minnow Support Project. It's a witness project run by aggroed, ausbitbank, teamsteem, theprophet0, someguy123, neoxian, followbtcnews/crimsonclad, and netuoso. The goal is to help Steemit grow by supporting Minnows and creating a social network. Please find us in the Peace, Abundance, and Liberty Network (PALnet) Discord Channel. It's a completely public and open space to all members of the Steemit community who voluntarily choose to be there.

This post has received a 1.56 % upvote from @drotto thanks to: @banjo.

@banjo Have you been good to them girls?

No. @banjo is still ignoring me. We need you there @precise. Where are you???? haha

No you didn't and I am not a mister.

in the blockchain @beng05 hahaha take good care of banjo

I could talk about Israel, but I don't think programming is directly involved.

Congratulations @precise! You have completed some achievement on Steemit and have been rewarded with new badge(s) :

Award for the total payout received

Click on any badge to view your own Board of Honor on SteemitBoard.
For more information about SteemitBoard, click here

If you no longer want to receive notifications, reply to this comment with the word STOP

By upvoting this notification, you can help all Steemit users. Learn how here!

Coin Marketplace

STEEM 0.30
TRX 0.12
JST 0.034
BTC 64455.55
ETH 3147.84
USDT 1.00
SBD 3.94