Re: Some Odd behavior on RHEL 5.4 with HTTPD (apache)

[Martin Pala]

Hi,

the httpd processes which keep running could be apache childs which are left to 
finish the pending requests when you stop apache. You can verify that by 
checking their PPID - if it's 1 (init), then they'll die as soon as they'll 
finish the work.

Have you verified that the stop+start commands work via Monit? Monit executes 
the start/stop/restart programs in a sandbox for security reasons and drops all 
environment variables and sets only spartan PATH=/bin:/usr/bin:/sbin:/usr/sbin. 
If your RHEL54-apache script depends on some environment variables, then it may 
fail.

You can verify the start/stop scripts this way:

        monit stop RHEL54-apache
        monit start RHEL54-apache 

You can also log the scripts output - modify the monit configuration file and 
reload monit:

  check process RHEL54-apache with path /etc/httpd/run/httpd.pid
    start program = "/bin/bash -c '/etc/init.d/httpd start 
>>/tmp/RHEL54-apache.log 2>&1'"
    stop program = "/bin/bash -c '/etc/init.d/httpd stop 
>>/tmp/RHEL54-apache.log 2>&1'"
    ...

Regards,
Martin



On Jan 13, 2013, at 11:53 AM, Bill G. <address@hidden> wrote:

> Hi List,
> 
> I may have stumbled across a bug, or I am just a nub (but I am pretty
> sure it is the former)
> 
> I am running RHEL 5.2 and 5.4 (dont ask why, just know that at this
> time it s a requirement)
> 
> For all intents and purposes the configurations between the two are
> identical (changing only hostnames and IP addresses)
> 
> Here is the first configuration (note all hostnames, ipaddresses, and
> service names are obfusticated):
> 
>   check host RHEL54-vip with address 10.0.0.1
>     start program = "/sbin/ifup eth0:1"
>     stop program = "/sbin/ifdown eth0:1"
>     if failed icmp type echo count 3 with timeout 3 seconds then restart
>     if 5 restarts within 5 cycles then timeout
> 
>   check process RHEL54-apache with path /etc/httpd/run/httpd.pid
>     start program = "/etc/init.d/httpd start"
>     stop program = "/etc/init.d/httpd stop"
>     if failed host RHEL54-vip port 80
>       protocol HTTP request "/" then restart
>     if 5 restarts within 5 cycles then timeout
> 
> Everything works as expected in 5.2, but in 5.4 you get the feeling
> that the world is ending.
> 
> If I force a crash from RHEL52:
> 
> Logs:
> 
> ICMP echo response for 10.0.0.1 1/3 timed out -- no response within 3 seconds
> ICMP echo response for 10.0.01 2/3 timed out -- no response within 3 seconds
> ICMP echo response for 10.0.0.1 3/3 timed out -- no response within 3 seconds
> 'RHEL52-vip' failed ICMP test [Echo Request]
> 'RHEL52-apache' failed, cannot open a connection to INET[RHEL52-vip:80/] via 
> TCP
> 'RHEL52-apache' stop: /etc/init.d/httpd
> 'RHEL52-apache' start: /etc/init.d/httpd
> 'RHEL52-apache' started
> 'RHEL52-apache' process is running with pid 5072
> 
> Everything else comes up and is happy
> 
> Same scenario on RHEL54
> nds
> nds
> nds
> 'RHEL54-vip' failed ICMP test [Echo Request]
>  TCP
> 'RHEL54-apache' stop: /etc/init.d/httpd
> 'RHEL54-apache' start: /etc/init.d/httpd
> 'RHEL54-apache' failed to start
> 
> Lather rinse repeat X5
> 
> What I have found is happening is that either the httpd process is not
> being shut down properly, and/or monit is not waiting for a proper
> return from the shutdown.
> 
> When in this condition:
> 
> # /etc/init.d/httpd status
> httpd is stopped
> ps -auxwww|grep httpd
> Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
> root     22031  0.1  0.5 236080 11200 ?        Ss   10:07   0:00 
> /usr/sbin/httpd
> apache   22033  0.0  0.3 236212  6640 ?        S    10:07   0:00 
> /usr/sbin/httpd
> apache   22034  0.0  0.3 236212  6636 ?        S    10:07   0:00 
> /usr/sbin/httpd
> apache   22035  0.0  0.3 236212  6636 ?        S    10:07   0:00 
> /usr/sbin/httpd
> apache   22036  0.0  0.3 236212  6636 ?        S    10:07   0:00 
> /usr/sbin/httpd
> apache   22037  0.0  0.3 236212  6636 ?        S    10:07   0:00 
> /usr/sbin/httpd
> apache   22038  0.0  0.3 236212  6636 ?        S    10:07   0:00 
> /usr/sbin/httpd
> apache   22039  0.0  0.3 236212  6636 ?        S    10:07   0:00 
> /usr/sbin/httpd
> apache   22040  0.0  0.3 236212  6636 ?        S    10:07   0:00 
> /usr/sbin/httpd
> 
> 
> (i give it to you typo and all!)
> 
> Only way to fix is to issue a killall command
> killall httpd
> then it starts as perscribed.
> 
> 
> So I thought, maybe it is something silly with the binary, so I
> decided to compile from source.. same issue.
> 
> My current workaround is as follows:
> 
>   check process RHEL54-apache with path /etc/httpd/run/httpd.pid
>     start program = "/etc/init.d/httpd start"
>     stop program = "/etc/init.d/httpd stop"
>     if failed host RHEL54-vip port 80
>       protocol HTTP request "/" then restart
>     if 2 restarts within 2 cycles then exec "/usr/bin/killall httpd"
>     if 5 restarts within 5 cycles then timeout
> 
> Any ideas how to make this function properly?
> --
> Thanks,
> Bill G.
> address@hidden
> 
> --
> To unsubscribe:
> https://lists.nongnu.org/mailman/listinfo/monit-general