mail notification problem

[Pavel Urban]

Hello,

our backup mailserver gets overloaded from time to time, so I've setup a 
monit installation to let me know when something happens. Today I've 
received notification of service failure, but no recover.

Sep 16 14:19:40 backmx monit[28991]: SMTP: error receiving data -- 
Resource temporarily unavailable
Sep 16 14:19:40 backmx monit[28991]: 'backmx' failed protocol test 
[SMTP] at INET[10.7.9.12:25] via TCP
Sep 16 14:19:45 backmx monit[28991]: Cannot open a connection to the 
mailserver 'localhost:25' -- Bad file descriptor
Sep 16 14:19:45 backmx monit[28991]: Trying the next mail server 
'10.7.9.11:25'
Sep 16 14:21:53 backmx monit[28991]: 'backmx' connection passed to 
INET[10.7.9.12:25] via TCP
Sep 16 14:21:58 backmx monit[28991]: Sendmail: error receiving data from 
the mailserver 'localhost' -- Resource temporarily unavailable
Sep 16 14:21:58 backmx monit[28991]: Aborting event

What's wrong?

address@hidden root]# monit summary
The monit daemon 4.7 uptime: 6h 28m

Process 'sshd'                      running
Remote Host 'backmx'                online with all services
Process 'freshclam'                 running
Process 'postgrey'                  running
Device 'root_backmx_fs'             accessible
Device 'spool_backmx_fs'            accessible
Device 'log_backmx_fs'              accessible
System 'backmx.iol.cz'              running

address@hidden root]# egrep -v '^(#|$)' /etc/monitrc
set daemon  120           # Poll at 2-minute intervals
set logfile syslog facility log_daemon # Set syslog logging
set mailserver localhost, 10.7.9.11
set mail-format           # Set a default mail from-address for
  { from: address@hidden }  # all alert messages emitted by monit
set alert address@hidden
set httpd port 2812 and   # Make monit start it's web-server
     use address localhost # and only accept connection from localhost
     allow localhost       # allow localhost to connect to the server and
     allow admin:monit     # user 'admin' with password 'monit'
check process sshd with pidfile /var/run/sshd.pid
  start program = "/etc/init.d/sshd start"
  stop program  = "/etc/init.d/sshd stop"
  if failed host localhost port 22 protocol ssh
     then restart
  if 3 restarts within 5 cycles then timeout
if loadavg(15min) is greater than 13 for 5 cycles
  then exec "/root/postfix/scripts/poststats-mail"
check host backmx address 10.7.9.12
  if failed port 25 protocol smtp with timeout 10 seconds then alert
check process freshclam with pidfile /var/run/clamav/freshclam.pid
  start program = "/etc/init.d/freshclam start"
  stop program  = "/etc/init.d/freshclam stop"
  if 3 restarts within 5 cycles then timeout
check process postgrey with pidfile /var/run/postgrey.pid
  start program = "/etc/init.d/postgrey start"
  stop program  = "/etc/init.d/postgrey stop"
  if 3 restarts within 5 cycles then timeout
check device root_backmx_fs with path /dev/sda2
  if 5 restarts within 5 cycles then timeout
  if space usage > 80 % then alert
  if inode usage > 80 % then alert
  mode passive
check device spool_backmx_fs with path /dev/sdb1
  if 5 restarts within 5 cycles then timeout
  if space usage > 80 % then alert
  if inode usage > 80 % then alert
  mode passive
check device log_backmx_fs with path /dev/sda5
  if 5 restarts within 5 cycles then timeout
  if space usage > 80 % then alert
  if inode usage > 80 % then alert
  mode passive

address@hidden root]# monit -V
This is monit version 4.7
Copyright (C) 2000-2006 by the monit project group. All Rights Reserved.

address@hidden root]# uname -a
Linux backmx.iol.cz 2.4.21-40.ELsmp #1 SMP Thu Feb 2 22:22:39 EST 2006 
i686 i686 i386 GNU/Linux

--
***********************************************************************
Pavel Urban (address@hidden)
O2 system disaster
Telefonica O2 Czech Republic, a.s. - www.cz.o2.com
***********************************************************************
   Vegetables should not operate electronic equipment.
          Computer Stupidities, http://rinkworks.com/stupid/
***********************************************************************