How to Restart Docker Crashed Containers Using Monit

Posted By :Harsh Kumar |18th October 2020

Monit is a small open source monitoring utility tool for Unix and Linux. Monit helps to monitor any processes like a web server, any database, any daemon on a server, or any other application. So Monit basically checks those services continuously in different intervals on a Linux or Unix server and makes sure they are running and if those services are not running then it will attempt to restart those services.

 

So today we will be going to learn how we can monitor Docker containers so that Monit can restart them when they crash and get stopped. In the case of Docker, we will be creating some scripts for containers that use the “docker top” command to monitor them.

 

There are those phases here. 1st phase to install Monit on the server (ubuntu) and 2nd configuring Monit to restart the crashed containers.

Install Monit on Linux server

Step1. Install Monit

 

$ sudo apt-get update

$ sudo apt-get install monit

 

Step2. Start Monit Daemon

 

$ sudo monit

 

Step3. Configure Monit Web service. (Default port 2812)

 

$ sudo nano /etc/monit/monitrc

 

add below lines to this file /etc/monit/monitrc:

set httpd port 2812

allow admin:monit # use user 'admin' with password 'monit'

 

1st line will set the port number 2812 for the Monit web server.

2nd line will set a user and password to access the Monit dashboard/web portal at 2812.

 

Step4. Reload the Monit to get the changes.

 

$ sudo monit -t ##checks the syntax

$ sudo monit reload

 

Step5. Access monit on the dashboard on port 2812 or you can check the monit status also by

 

$ sudo monit status

 

Monit

 

Setup Monit to restart Docker containers automatically

 

So the main concept here is to monitor the container with the help of the "docker top" command.

We have to create a /etc/monit/scripts/check_container_<container-name>.sh file for the container we want to restart by Monit which is going to be this command in it.

 

#! /bin/bash
docker top "<container-name>"
exit $?

 

Replace <container-name> in both file’s name and the in file itself with the container you want to monitor. For example, a container named nginx the respective file name would be etc/monit/scripts/check_container_nginx.sh and the command in it would be like docker top “nginx”.

 

Then make .sh file executable my below command:

$ chmod +x /etc/monit/scripts/check_container_<container-name>.sh

 

Then create a file named check_container_<container-name> in /etc/monit/conf.d which has below content:

 

CHECK PROGRAM <container-name> WITH PATH /etc/monit/scripts/check_container_<container-name>.sh
  START PROGRAM = "/usr/bin/docker start <container-name>"
  STOP PROGRAM = "/usr/bin/docker stop <container-name>"
  IF status != 0 FOR 3 CYCLES THEN RESTART
  IF 2 RESTARTS WITHIN 5 CYCLES THEN UNMONITOR

 

Then check that in /etc/monit/monitrc this line “include /etc/monit/conf.d/*” is included and not commented out, and then restart the Monit with systemctl restart monit.


 

This setup monitors the Container and restarts it when stopped abruptly.


About Author

Harsh Kumar

He is an enthusiastic employee who has worked on cloud platforms like Azure and AWS and some DevOps tools.

Request For Proposal

Sending message..

Ready to innovate ? Let's get in touch

Chat With Us