How to Restart Docker Crashed Containers Using Monit

Monit is a small open source monitoring utility tool for Unix and Linux. Monit helps to monitor any processes like a web server, any database, any daemon on a server, or any other application. So Monit basically checks those services continuously in different intervals on a Linux or Unix server and makes sure they are running and if those services are not running then it will attempt to restart those services.

So today we will be going to learn how we can monitor Docker containers so that Monit can restart them when they crash and get stopped. In the case of Docker, we will be creating some scripts for containers that use the “docker top” command to monitor them.

There are those phases here. 1^st phase to install Monit on the server (ubuntu) and 2^nd configuring Monit to restart the crashed containers.

Install Monit on Linux server

Step1. Install Monit

$ sudo apt-get update

$ sudo apt-get install monit

Step2. Start Monit Daemon

$ sudo monit

Step3. Configure Monit Web service. (Default port 2812)

$ sudo nano /etc/monit/monitrc

add below lines to this file /etc/monit/monitrc:

set httpd port 2812

allow admin:monit # use user 'admin' with password 'monit'

1st line will set the port number 2812 for the Monit web server.

2nd line will set a user and password to access the Monit dashboard/web portal at 2812.

Step4. Reload the Monit to get the changes.

$ sudo monit -t ##checks the syntax

$ sudo monit reload

Step5. Access monit on the dashboard on port 2812 or you can check the monit status also by

$ sudo monit status

Setup Monit to restart Docker containers automatically

So the main concept here is to monitor the container with the help of the "docker top" command.

We have to create a /etc/monit/scripts/check_container_<container-name>.sh file for the container we want to restart by Monit which is going to be this command in it.

#! /bin/bash
docker top "<container-name>"
exit $?

Replace <container-name> in both file's name and the in file itself with the container you want to monitor. For example, a container named nginx the respective file name would be etc/monit/scripts/check_container_nginx.sh and the command in it would be like docker top “nginx”.

Then make .sh file executable my below command:

$ chmod +x /etc/monit/scripts/check_container_<container-name>.sh

Then create a file named check_container_<container-name> in /etc/monit/conf.d which has below content:

CHECK PROGRAM <container-name> WITH PATH /etc/monit/scripts/check_container_<container-name>.sh
  START PROGRAM = "/usr/bin/docker start <container-name>"
  STOP PROGRAM = "/usr/bin/docker stop <container-name>"
  IF status != 0 FOR 3 CYCLES THEN RESTART
  IF 2 RESTARTS WITHIN 5 CYCLES THEN UNMONITOR

Then check that in /etc/monit/monitrc this line “include /etc/monit/conf.d/*” is included and not commented out, and then restart the Monit with systemctl restart monit.

This setup monitors the Container and restarts it when stopped abruptly.