When the Client Reports the Production is Down at Mid Night

in Steem Sri Lanka2 months ago


It was Sunday night. On Saturday night I upgraded the production environment with the latest version and the client verified the system working properly on Sunday morning. I spend the day so freely and went to the bed around 11 p.m.

At the same time, I got a call from one of our developers, the system can not be accessed. I quickly check out the environment and noticed the system is completely down! It is a system of a finance company and the system must up before Monday morning.

I was just confused because it is still 3 months since I have started my job and still, I am unable to figure out architectural issues. The environment is Kubernetes and the following was the error.

When kubectl get pods -o wide
The connection to the server <apiserver_advertise_ip>:6443 was refused - did you specify the right host or port?

I opened a google meet with two other Devops Engineers and tried to figure out the cause. First, we thought it is an error due to closing the ports. Also, we tried restarting kubelet service but couldn't come to a proper solution until 1.00 a.m.


At 1.00 a.m we just checked the kubelet logs and realized all happened because the Kubernetes certificate is expired. Kubernetes had to tell us this in a proper way. We were using v1.14.10 so the certificate updating procedure was pretty long.

It was almost 2.00 a.m when we find the steps to update certificates. There were four of us and we didn't forget to keep in contact with the client who started to test production at 11.00 p.m :).

Following blog made by one of our early developers was very useful to us to finish the task. At 3.30 a.m we were able to up the production environment back.

I was almost hungry and sleepy. The best lesson, I made a calendar reminder to update the Kubernetes certificate before it expires next year.



Hola! It is our lovely friend @alejos7ven 's birthday..


Ale is a dedicated, friendly, and very honest person who works hard to achieve his goals. He is a vampire too. We met recently but became friends fast.

Join with me to wish a very happy birthday to @alejos7ven and a good future!


THANK YOU SO MUCH LITTLE FOR YOUR AMAZING MENTION 🥺 and the beneficiary, you are one of the best friends what I meet in this platform, is a pleasure talk with you and other guys everyday :)

Thanks to you and @psicoparedes and all Venezuela friends for being such lovely.

@randulakoralage oh, thank you so much for taking me into consideration 😊 Venezuela has got many friendly people. At least we all have come across and been able to meet each other. I am glad to have met you both and the rest of Sri Lankan friends. You are such amazing partners

I was almost hungry and sleepy.

I can understand this feeling. I have that experience too 😥

Getting old :(

@randulakoralage That is the nature of jobs .... Be happy with codes.... 😜😜😜......

I am a person who slept at 10.00. Oh my life

At 3.30 a.m we were able to up the production environment back.

I can imagine how exhausted you are 😥.

yeah.. It is so stressfull

 2 months ago 

Your post has been upvoted by Steem Sri Lanka Community Curation Trail.

Steem Sri Lanka Discord Channel


Join With
Curation Trail For Better Curation Rewards