Re: Monit Issue during reboot

[Geoff Goas]

What if you setup an entry in inittab to kill monit when it switches to runlevel 0 or 6? Making the assumption of course that runlevel 0 is halt and 6 is reboot.

On Mon, Nov 21, 2016 at 3:13 PM, Shankar SM <address@hidden> wrote:

Hi Geoff,

        The services are actually started by init script, so during shutdown the kill scripts are called.
Since monit is started from inittab, there is a chance that monit daemon interval expires during a shutdown process and restarts applications that have shutdown gracefully.
So basically it is to ensure a clean shutdown of all services.

Thanks
Shankar


From: monit-general [mailto:monit-general-bounces+ssm=address@hidden] On Behalf Of Geoff Goas
Sent: Monday, November 21, 2016 2:19 PM
To: This is the general mailing list for monit <address@hidden>
Subject: Re: Monit Issue during reboot

What is the reason for having monit unmonitor all services prior to shutdown?

On Mon, Nov 21, 2016 at 8:51 AM, Shankar SM <address@hidden> wrote:
Hi All,
                We are seeing an issue using monit during a reboot or shutdown sequence. First some background of the some of the configuration of monit.

1. Monit version is 5.5
2. Monit is started from inittab as null::respawn:/usr/bin/monit -Ic /etc/monitrc 3. Monit is configured to check services every 30 secs.
4. Monit is monitoring certain custom applications and restarts them if they are not running by calling the init script of that particular custom application.
5. The applications have their own init script in /etc/init.d/ folder that are started on bootup. Monit starts after these init scripts are run from inittab.

Now to the problem at hand.
During a shutdown/reboot of the system first a command is sent to monit to stop monitoring all services. After this all the kill scripts in /etc/init.d/ are called. For the most part it works but there is a corner case where monit starts an application when the application has already shutdown gracefully, which is wrong. I have tried to capture the problem in the Sequence diagram below link.

https://i.stack.imgur.com/XwYk9.png

I looked at monit source code and it looks like when the check for a service fails monit tries to run the command mentioned by forking and executing it. So this forked process can still be in progress when monit receives a unmonitor command. It also looks like monit does not stop processes that it started when it receives this command and immediately returns.

Is there a way to wait till all the forked processes that are started by monit are complete?

Any other recommendation to avoid this problem?

Thanks
Shankar


--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general




--
Geoff Goas
Systems Engineer
--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

--

Geoff Goas
Systems Engineer