[RFD] monit method definition language (present start and stopprogram extension) + globalization

[Martin Pala]

Request for Discussion: 
Theme: monit method definition language (present start and stop program extension)

Aim of this mail is to discuss and figure out syntax (and hopefully entry 
setting) for monit optional method execution extension.
All users are welcome to give us their feedback and participate on monit 
evolution :)


The goal of this extension is to support definition of methods (i.e. programs) 
which will monit call/execute in the case that some monitored parameter match 
the condition.

Present version of monit understands "start" and "stop" programs which are used 
for corresponding actions or are chained for the purpose of monitored service restart. Dependency 
can be optionaly defined between monitored services and monit will do what is needed to keep the 
rigth start/stop order for the services.

This system is very simple, however is limited for these atomic operations. 
There are conditions which can be better cared by optional methods depending on 
the service type.


Proposal:

1a.) generalizing syntax of monitored service's method definition

Present syntax:

CHECK ...
START [PROGRAM] [=] "/path/to/program parameters ..."
STOP  [PROGRAM] [=] "/path/to/program parameters ..."

Real world example:

check process apache with pidfile /usr/local/apache/logs/httpd.pid
start program = "/etc/init.d/httpd start"
stop program  = "/etc/init.d/httpd stop"


New (general) syntax:

METHOD name [=] "/path/to/program parameters ..."

Real world example according to above present syntax example - it is shown as 
local method definition (inside of CHECK statement - see section 2b of this 
proposal for global method proposal):

check process apache with pidfile /usr/local/apache/logs/httpd.pid
method start = "/etc/init.d/httpd start"
method stop  = "/etc/init.d/httpd stop"


Until here it provides even functionality as present syntax. Benefit of new 
syntax is support for additional methods definition:

method shutdown "/sbin/shutdown -t 1 bye, bye"
method reload   "/bin/kill -HUP $MONIT_PROGRAM_PID"
method foo      "/path/to/foo -a opt1 -b opt2 opt3"

Methods which were defined this way can be used (by its name) in optional 
"action" field (explained bellow) of the particular test, for example:

if timestamp of "/etc/httpd/conf/httpd.conf" changed then reload

It is possible to chain methods in the action field by defining coma separated 
list of methods (note - following example will reuse atomic stop and then start 
methods for the purpose of service restart):

if timestamp of "/etc/httpd/conf/httpd.conf" changed then stop, start


Action field has following syntax:

IF TEST MATCH CONDITION THEN ACTION

where ACTION field matches any service's method name OR special built-in method 
named 'alert' (which is currently the only monit's built-in method and produces 
e-mail message).

Example1:

if timestamp of "/etc/httpd/conf/httpd.conf" changed then reload
# it will exec method named 'reload' and produce usual 'timestamp' alert

Example2:
if timestamp of "/etc/httpd/conf/httpd.conf" changed then alert
# it will produce just alert


Monit gives the method informations for the event in environment variables, 
current development monit version supports following (any recommendations for 
extension are welcomed):
MONIT_DATE
MONIT_HOST
MONIT_PROGRAM
MONIT_PROGRAM_PID
MONIT_PROGRAM_MEMORY
MONIT_PROGRAM_CHILDREN
MONIT_PROGRAM_CPU_PERCENT
#MONIT_EVENT
#MONIT_SERVICE
#MONIT_ACTION

In the case that you define method named 'alert', it will be used instead of 
monit's default (it overrides it - same rules are valid for global and local 
method conflict resolution described in section 2b bellow). For example:

method alert "/send/sms -u address@hidden -u address@hidden 
$MONIT_HOST:$MONIT_DATE:$MONIT_PROGRAM:$MONIT_EVENT"

The rules for such method usage are the same as for any other method (so it can use 
environment variables). Monit alert list (for example defined by "alert 
address@hidden on { timeout }") is used only by built-in alert mechanism.


1b.) Alert information extension

Default alert mechanism will be modified to describe method's name (action) 
which was executed (including special case 'alert').

New default alert message:

---
Service $SERVICE $EVENT

 Date:   $DATE
 Host:   $HOST
 Action: $ACTION  <-------- NEW

Your faithful employee,
monit
---

Note: the mechanism for alert event registration is still valid, so you can 
choose just subset of error messages for particular users. If you register no 
user for alert messages, then no messages will be delivered.


1c.) Method dependency modification

There are two ways to care for services dependency in the case of error event:

- either push the same method that was executed for the service which produced 
the event down the dependency tree in all cases (equals to present monit 
behavior)
- or push by default as above but in addition support private service methods 
types, which won't be executed be dependants

Example1:
method start = "/etc/init.d/httpd start"
# the method named 'start' will be executed on all dependants down the tree 
(only name is pushed - method body is service dependent)

Example2:
method reload = "/etc/init.d/httpd reload" type private
# the method named 'reload' will be executed just for the service where it was 
defined in the case that it was called. The built-in method 'alert' could be 
example of such method.

Because it is not allways usefull to push all action types to dependants (for 
example on service configuration reload it may not be needed to reload whole 
chain), i'm more inclined to add (optional) private methods support.


2.) global method and alert statements definition extension

Global statements are proposed to support definition of statements usage which 
are common for all services and occures in all service instance.

Such statements are for example ALERT:

alert address@hidden

and METHOD:

method reload "/bin/kill -HUP $MONIT_PROGRAM_PID"

They can be usualy reused in all monitored services => we can inherite them 
from global definition.


2a.) global alert

Syntax:

---
set alert address@hidden 
set alert address@hidden on { checksum }

check ...
alert address@hidden
---

Global alert statements are defined before 'check ...' statements and they 
start with 'set' keyword. More then one instance of 'set alert ...' is allowed, 
it comprises global alert list.

All services inherites global list, in addition they can define local alert 
targets. As result monit will send message to the list of users which is 
compiled from global and local alert statements (only users which registered 
this type  of event are sent the message).

In the case that the user is defined in global and local list, it is sent only 
one message and for the decision whether he/she has registered this event is 
used local definition (local selection overrides global). For example:

---
set alert address@hidden on { checksum }

check ...
 alert address@hidden
---

=> for all monitored services only checksum error is sent to address@hidden, 
with the exception of service defined in the statement 'check ...', for which all 
errors will be sent to the given address.


2b.) global method

Syntax:

---
set method shutdown "/sbin/shutdown -t 1 bye, bye"
set method reload   "/bin/kill -HUP $MONIT_PROGRAM_PID"

check ...
method start = "/etc/init.d/httpd start"
---

Global method statements are defined before 'check ...' statements and they 
start with 'set' keyword. More then one instance of 'set method ...' is 
allowed, it comprises global method list.

Global methods are inherited and accessible by all monitored services. In 
addition they can define local methods.

In the case that the method of the same name is defined in global and local 
list, local instance has precedence (overrides global method). For example:

---
set method reload   "/bin/kill -HUP $MONIT_PROGRAM_PID"

check ...
 method reload   "/etc/init.d/program reload"
---

=> for all monitored services default 'reload' method is inherited, with the exception of 
service defined in the statement 'check ...', for which local method 
("/etc/init.d/program reload") will be executed.




Cheers,
Martin