Cron and heartbeat monitor

The way heartbeat monitoring works is that you need to periodically make requests to a monitor's unique URL in order for the monitor NOT to create a new incident.

Example usage

We have a background job that makes a daily database backup of our primary PostgreSQL database. We want to get alerted if the background job doesn't run successfully.

Creating a heartbeat monitor:

  • Go to IntegrationsCreate heartbeat.
  • Name your heartbeat — "Daily database backup".
  • Change the Expect a heartbeat every selection to "1 day".
  • Optionally configure the On-call escalation settings.
  • Click Save heartbeat.
  • Copy the secret URL on the Heartbeat detail page; you will need it later.

This heartbeat will expect us to make a GET or POST request to the URL provided to us every day after the first request.

Set up a CRON and a background job

  1. Add a CRON task that will execute backup_database.sh background job every day at midnight:

     
    0 0 * * * ruby /home/deploy/backup_database.sh >/dev/null 2>&1
    
  2. Include the curl call to the heartbeat URL at the end of your backup script:

     
    #!/usr/bin/env bash
    
    set -o errexit
    set -o xtrace
    
    date=`date "+%Y-%m-%d_%H:%M:%S"`
    file="/dumps/uptime.betterstack.$date.dump"
    
    time dokku postgres:export uptime > "$file"
    
    /usr/local/bin/aws s3 cp "$file" s3://uptime-dbbackups/
    
    rm "$file"
    
    # you get this URL in the Uptime dashboard
    curl "https://uptime.betterstack.com/api/v1/heartbeat/XXXXXXXXXXXXXXXXXXXXXX"
    

What happens here is that the Heartbeat URL we've created above expects a GET or POST request every day since having made the first request.

If the code above fails, our background job won't make the request to the Heartbeat URL. In that case, the Heartbeat will alert the current on-call person and create an Incident.