|
| From: | Bill G. |
| Subject: | Re: Some Odd behavior on RHEL 5.4 with HTTPD (apache) |
| Date: | Mon, 14 Jan 2013 06:33:28 -0800 |
Hi Martin,
I know for a fact that they are not child processes, as the parent process is still alive, and the init script doesn't recognize HTTPS as running.
What I am theorizing is that monit is running the restart processes simultaneously, i.e. instead of /etc/init.d/httpd stop .....waiting for stop.....stopped now issue the start command /etc/init.d/httpd start.
It is issuing the commands as two separate threads and this is confusing the init scripts possibly because it generally takes longer for HTTPD to stop then start. So it issues the stop and start at the same time. Stop wins the race as the first process to start, initializes the signal to kill waits, while next to hit is the httpd start command it says oh look pid file is there but the pid it lists is gone, OK start, start successful. Meanwhile the stop command finishes and removes the pid file. To top it off when it tries to restart it fails to start due to the fact that port 80 is indeed listening and apache is running, the init system is, however, unaware.
I hope that makes sense.
Thanks,
Bill
Hi,
the httpd processes which keep running could be apache childs which are left to finish the pending requests when you stop apache. You can verify that by checking their PPID - if it's 1 (init), then they'll die as soon as they'll finish the work.
Have you verified that the stop+start commands work via Monit? Monit executes the start/stop/restart programs in a sandbox for security reasons and drops all environment variables and sets only spartan PATH=/bin:/usr/bin:/sbin:/usr/sbin. If your RHEL54-apache script depends on some environment variables, then it may fail.
You can verify the start/stop scripts this way:
monit stop RHEL54-apache
monit start RHEL54-apache
You can also log the scripts output - modify the monit configuration file and reload monit:
check process RHEL54-apache with path /etc/httpd/run/httpd.pid
start program = "/bin/bash -c '/etc/init.d/httpd start >>/tmp/RHEL54-apache.log 2>&1'"
stop program = "/bin/bash -c '/etc/init.d/httpd stop >>/tmp/RHEL54-apache.log 2>&1'"
...
Regards,
Martin
On Jan 13, 2013, at 11:53 AM, Bill G. <address@hidden> wrote:
> Hi List,
>
> I may have stumbled across a bug, or I am just a nub (but I am pretty
> sure it is the former)
>
> I am running RHEL 5.2 and 5.4 (dont ask why, just know that at this
> time it s a requirement)
>
> For all intents and purposes the configurations between the two are
> identical (changing only hostnames and IP addresses)
>
> Here is the first configuration (note all hostnames, ipaddresses, and
> service names are obfusticated):
>
> check host RHEL54-vip with address 10.0.0.1
> start program = "/sbin/ifup eth0:1"
> stop program = "/sbin/ifdown eth0:1"
> if failed icmp type echo count 3 with timeout 3 seconds then restart
> if 5 restarts within 5 cycles then timeout
>
> check process RHEL54-apache with path /etc/httpd/run/httpd.pid
> start program = "/etc/init.d/httpd start"
> stop program = "/etc/init.d/httpd stop"
> if failed host RHEL54-vip port 80
> protocol HTTP request "/" then restart
> if 5 restarts within 5 cycles then timeout
>
> Everything works as expected in 5.2, but in 5.4 you get the feeling
> that the world is ending.
>
> If I force a crash from RHEL52:
>
> Logs:
>
> ICMP echo response for 10.0.0.1 1/3 timed out -- no response within 3 seconds
> ICMP echo response for 10.0.01 2/3 timed out -- no response within 3 seconds
> ICMP echo response for 10.0.0.1 3/3 timed out -- no response within 3 seconds
> 'RHEL52-vip' failed ICMP test [Echo Request]
> 'RHEL52-apache' failed, cannot open a connection to INET[RHEL52-vip:80/] via TCP
> 'RHEL52-apache' stop: /etc/init.d/httpd
> 'RHEL52-apache' start: /etc/init.d/httpd
> 'RHEL52-apache' started
> 'RHEL52-apache' process is running with pid 5072
>
> Everything else comes up and is happy
>
> Same scenario on RHEL54
> nds
> nds
> nds
> 'RHEL54-vip' failed ICMP test [Echo Request]
> TCP
> 'RHEL54-apache' stop: /etc/init.d/httpd
> 'RHEL54-apache' start: /etc/init.d/httpd
> 'RHEL54-apache' failed to start
>
> Lather rinse repeat X5
>
> What I have found is happening is that either the httpd process is not
> being shut down properly, and/or monit is not waiting for a proper
> return from the shutdown.
>
> When in this condition:
>
> # /etc/init.d/httpd status
> httpd is stopped
> ps -auxwww|grep httpd
> Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
> root 22031 0.1 0.5 236080 11200 ? Ss 10:07 0:00 /usr/sbin/httpd
> apache 22033 0.0 0.3 236212 6640 ? S 10:07 0:00 /usr/sbin/httpd
> apache 22034 0.0 0.3 236212 6636 ? S 10:07 0:00 /usr/sbin/httpd
> apache 22035 0.0 0.3 236212 6636 ? S 10:07 0:00 /usr/sbin/httpd
> apache 22036 0.0 0.3 236212 6636 ? S 10:07 0:00 /usr/sbin/httpd
> apache 22037 0.0 0.3 236212 6636 ? S 10:07 0:00 /usr/sbin/httpd
> apache 22038 0.0 0.3 236212 6636 ? S 10:07 0:00 /usr/sbin/httpd
> apache 22039 0.0 0.3 236212 6636 ? S 10:07 0:00 /usr/sbin/httpd
> apache 22040 0.0 0.3 236212 6636 ? S 10:07 0:00 /usr/sbin/httpd
>
>
> (i give it to you typo and all!)
>
> Only way to fix is to issue a killall command
> killall httpd
> then it starts as perscribed.
>
>
> So I thought, maybe it is something silly with the binary, so I
> decided to compile from source.. same issue.
>
> My current workaround is as follows:
>
> check process RHEL54-apache with path /etc/httpd/run/httpd.pid
> start program = "/etc/init.d/httpd start"
> stop program = "/etc/init.d/httpd stop"
> if failed host RHEL54-vip port 80
> protocol HTTP request "/" then restart
> if 2 restarts within 2 cycles then exec "/usr/bin/killall httpd"
> if 5 restarts within 5 cycles then timeout
>
> Any ideas how to make this function properly?
> --
> Thanks,
> Bill G.
> address@hidden
>
> --
> To unsubscribe:
> https://lists.nongnu.org/mailman/listinfo/monit-general
--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general
| [Prev in Thread] | Current Thread | [Next in Thread] |