[monit] Problem monitoring mongrel Rails servers

[SMITH Duncan]

Hello,

I am having difficulty monitoring a cluster of mongrel servers on Solaris.

I have a cluster of mongrel servers, and my .monitrc script looks like this (there is a similar entry for each mongrel server, with port number changed):

 check process mongrel_prod_30000 with pidfile /web/docs/forms.eurostar.inet/current/tmp/pids/mongrel.30000.pid
  if failed host elbrus port 30000 protocol http
     and request "/" then alert
  if totalmem > 100 Mb then restart
  if cpu > 60% for 2 cycles then alert
  if cpu > 80% for 5 cycles then restart
  if loadavg(5min) greater than 10 for 8 cycles then restart
  if 3 restarts within 5 cycles then timeout

  start program = "/usr/local/bin/mongrel_rails cluster::start -C /web/docs/forms.eurostar.inet/current/config/clusters/production.yml --only 30000 --clean"

  stop program = "/usr/local/bin/mongrel_rails cluster::stop -C /web/docs/forms.eurostar.inet/current/config/clusters/production.yml --only 30000 --force"

If I run monit everything seems fine, I see all the mongrel servers running. If I kill one of them, for example the one referred to above, I get an email alert:

Does not exist Service mongrel_prod_30000

And the web page sets the mongrel to "not monitored". I then get another email alert:

Execution failed Service mongrel_prod_30000

        Date:        Thu, 03 Apr 2008 16:02:51 +0100
        Action:      alert
        Host:        elbrus
        Description: 'mongrel_prod_30000' failed to start

Your faithful employee,
monit

I have checked for permissions problems, but the start and stop scripts work fine from the commandline, run as root, and the monit is running as root also.

I also tried monitoring apache, and the start/stop worked fine using a similar script, so I expect something may be wrong with mongrels but I cannot see what it could be, since they work fine otherwise.

I don't want to have to monitor my monitor!

Any advice warmly received!

Duncan Smith

Important Note: This E-Mail is confidential, it must not be read, 
copied, disclosed or used by any person other than the above
named addressees. Unauthorised use, disclosure or copying is
strictly prohibited and may be unlawful. Eurostar (U.K.) Limited
and Eurostar Group Limited disclaim any liability for any action
taken in reliance on the content of this E-Mail. The comments
or statements expressed in this E-Mail are not necessarily those
of Eurostar (U.K.) Limited, Eurostar Group Limited or any
subsidiaries or affiliates of either company.
Eurostar (U.K.) Ltd. Registered Office: Eurostar House, Waterloo
Station, London, SE1 8SE, Registered in England No. 2462001
VAT No. GB 657 719 590
Eurostar Group Ltd. Registered office: Eurostar House, Waterloo
Station, London, SE1 8SE, Registered in England No. 3794642
VAT No. GB 739 974 173

This email has been scanned for all viruses by the MessageLabs Email
Security System on behalf of Eurostar. For more information on a proactive email security service working around the clock, around the globe, visit http://www.messagelabs.com