Re: pagerduty integration with M/Monit

[Paul Theodoropoulos]

On 2/10/15 1:12 PM, Christopher Wood wrote:
> Maybe instead of using their wrapper, see how you can call the pagerduty 
> binary/script directly?
>
> /usr/local/bin/pagerduty
>
> https://github.com/pinterest/pagerduty-monit/blob/master/pagerduty-trigger
>
> In that script:
>
> /usr/local/bin/pagerduty -k "$PAGERDUTY_SERVICE_KEY" -i "$INCIDENT_KEY" 
> --description="$DESCRIPTION" trigger
>
> The $DESCRIPTION is just
>
> EVENT="$1"
> HOST=`hostname -s`
> DESCRIPTION="$EVENT failed on $HOST"
>
> You can likely produce the message you want as long as you can pass the remote 
> hostname from mmonit. (As to how, sorry, only used monit itself.)
Yeah, all it required was minor massaging of their wrapper - and pasting 
the wrapper script ito the 'execute command' field, rather than calling 
the script via the field. For anyone else who might wish to integrate 
PD, here's the mods (which can be further modded, obviously)

pagerduty-trigger:

#!/bin/bash
#
# pagerduty-trigger is a wrapper around the PagerDuty API to open a new
# incident from the command line. This script is designed to help integrate
# monit with PagerDuty.
#
# Copyright (c) 2012 Cold Brew Labs, Inc. See LICENSE for details.
#
# Uncomment and enter your service key here.
PAGERDUTY_SERVICE_KEY="abc123youandme"

#if [ -z "$1" ]; then
#  echo "Usage: $0 <event>"
#  exit 1
if [ -z "$PAGERDUTY_SERVICE_KEY" ]; then
  echo "Failed to trigger event: you must set the PAGERDUTY_SERVICE_KEY 
variable."
  exit 1
elif [ ! -x "/usr/local/bin/pagerduty" ]; then
  echo "Failed to trigger event: /usr/local/bin/pagerduty does not 
exist or is not executable."
  exit 1
fi

#EVENT="$1"
#HOST=`hostname -s`
INCIDENT_KEY=`echo "$MONIT_HOST:$MONIT_SERVICE" | md5sum | cut -f 1 -d ' '`
TMP_FILE="/tmp/pagerduty-$INCIDENT_KEY"

if [ -f "$TMP_FILE" ]; then
  # re-trigger after 4 hrs 1 min (the extra minute is to ensure the 
incident auto-resolved with PagerDuty)
  if [ "$(( $(date +"%s") - $(stat -c "%Y" $TMP_FILE) ))" -lt "14460" 
]; then
    echo "$TMP_FILE exists, aborting"
    exit 0
  else
    echo "$TMP_FILE exists but is older than 4 hours; re-triggering"
  fi
fi

DESCRIPTION="$MONIT_SERVICE $MONIT_EVENT $MONIT_HOST"
DATE=`date`

echo "$DATE: $DESCRIPTION" >> $TMP_FILE
/usr/local/bin/pagerduty -k "$PAGERDUTY_SERVICE_KEY" -i "$INCIDENT_KEY" 
--description="$DESCRIPTION" trigger

if [ "$?" -ne "0" ]; then
  echo "Failed to trigger incident"
  exit 1
fi

echo "Incident triggered successfully"
exit 0

=======================================
pagerduty-resolve:

#!/bin/bash
#
# pagerduty-resolve is a wrapper around the PagerDuty API to close an 
incident
# from the command line.
#
# Copyright (c) 2012 Cold Brew Labs, Inc. See LICENSE for details.

# Uncomment and enter your service key here.
PAGERDUTY_SERVICE_KEY="abc123youandme"

#if [ -z "$1" ]; then
#  echo "Usage: $0 <event>"
#  exit 1
if [ -z "$PAGERDUTY_SERVICE_KEY" ]; then
  echo "Failed to resolve event: you must set the PAGERDUTY_SERVICE_KEY 
variable."
  exit 1
elif [ ! -x "/usr/local/bin/pagerduty" ]; then
  echo "Failed to resolve event: /usr/local/bin/pagerduty does not 
exist or is not executable."
  exit 1
fi

#EVENT=$1
#HOST=`hostname -s`
INCIDENT_KEY=`echo "$MONIT_HOST:$MONIT_SERVICE" | md5sum | cut -f 1 -d ' '`
TMP_FILE="/tmp/pagerduty-$INCIDENT_KEY"

DESCRIPTION="$MONIT_SERVICE $MONIT_EVENT $MONIT_HOST"

rm -f $TMP_FILE
/usr/local/bin/pagerduty -k "$PAGERDUTY_SERVICE_KEY" -i "$INCIDENT_KEY" 
--description="$DESCRIPTION" resolve

if [ "$?" -ne "0" ]; then
  echo "Failed to resolve incident"
  exit 1
fi

echo "Incident resolved successfully"
exit 0

--
Paul Theodoropoulos
www.anastrophe.com