Notification Retry and Retention

Insuring delivery of notifications is important. To provide guaranteed delivery, the system provides a delivery retry mechanism as well as retention policy rules for how long notifications are saved for historical retrieval. The system will attempt to deliver the notifications using an exponential backoff algorithm until the delivery queue is full. At that point the delivery queue is flushed and undelivered messages can then be retrieved from the historical retention.

Configuration

Retry and notification delivery options are configured during Tenant creation. Detailed steps in the Potal Guide - Manage Tenants.

Here are the notificaiton specific configuration parameters.


Parameter     

Description

Valid Values

Queue Retention Policy

Defines if the queue retention policy is priority based or size based. If the queue reaches the retention limit, set in Queue Retention limits, this determines if the messages will be replaced based on priority or time (keep the most recent and remove the oldest)

None, PriorityBased, TimeBased

Queue Retention Size

Size limit of the notification queue in GB if using size based retention.

.5, 1, 2, 3

Queue Retention Time

Maximum duration in hours that the system will keep a notification in the notification queue. When this time is reached, all notifications in the queue longer than this time will be removed from the delivery queue and must be retrieved from the historical retention via API call. (see /api/v1/events/customerevent/history)

1, 3, 6, 9, 12, 24

Customer Event Retention Time

Maximum time in hours that the system will keep a notification in permanent storage. When this time is reached, all notifications in the storage longer than this time will be removed and will no longer be available for retrieval via the API.

12 ,24, 48

Webhook Retry Duration

Maximum time in hours that the Notification Service will attempt to deliver to a webhook endpoint. After this time the delivery is no longer attempted and must be retrieved from via the API. Once the Customer Event Retention Time is reached, it will be removed from the queue.

.5, 1, 2, 3, 5

Required Queue Notification

Access to read only Service Bus details to read queue values.

Yes, No


Webhook Retry

Notifications configured for Webhook delivery will use a back-off retry with jitter algorithm.

The algorithm is:

  • Min is for 5 seconds and max is 10 minutes
  • After failure Retry at min interval (5 seconds in this case).
  • Retry min interval + random jitter (20% of min)
  • Retry min interval + random jitter (20% of min)
  • Intentionally do this twice
  • Then start backing off with 20% jitter
  • 10 seconds + 20% jitter
  • 15 seconds + 20% jitter 
  • 30 seconds + 20% jitter
  • 65 seconds + 20% jitter
  • ...
  • Until max is reached
  • Then continue at max + 20% jitter until Webhook Retry Duration is reached
  • Once Webhook Retry Duration is reached, the notification is marked un-delivered in the historical retention and dropped from delivery queue.


Webhook Retry Example


      Assume these system settings:

  • Queue Retention Time = 3 hours
  • Customer Event Retention Time = 12 hours
  • Webhook Retry Duration = 1 hour
  • System is configured to deliver an attribute output notification.
  1. 9:00am attribute event #1 arrives from the device
    1.  Notification #1 is stored in historical retention
    2. The webhook endpoint is available
      1. Notification #1 is successfully delivered
      2. Notification #1 is marked read in historical retention
  2. 9:10am attribute event #2 arrives from the device
    1. Notification #2 is stored in the Historical Retention
    2. The webhook endpoint is unavailable.
      1. Notification service starts retry algorithm for #2.
  3. 9:15am Webhook endpoint comes back up
    1. Notification #2 is delivered
    2. notification #2 is marked read in historical retention
  4. 9:20am Attribute event #3 arrives from the device
    1. Notification #3 is stored in the Historical Retention
    2. The webhook endpoint is unavailable.
      1. Notification service starts retry algorithm for #3.
  5. 10:20am the webhook endpoint is still down
    1. Webhook service stops trying to deliver notification #3
    2. Notification #3 is marked unread in historical retention
  6. 10:25am webhook endpoint is available
    1. Since webhook retry has expired, notification #3 is not delivered.
  7. 10:30am Attribute event #4 arrives from the device
    1. Notification #4 is stored in historical retention
    2. The webhook endpoint is available
      1. notification #4 is successfully delivered
      2. notification #4 is marked read in the historical retention
  8. 12:20pm – Queue Retention time check is run
    1. Notification #3 is removed from the queue since it is more than 3hours old.
  9. 1:30pm – API call to retrieve historical retention
    1. Results


Notification #1: read
Notification #2: read
Notification #3: unread
Notification #4: read


  1. 9:00pm – Historical retention check is run
    1. Notification #1 removed from retention since it is more than 12 hours old.
  2. 9:05pm API call to retrieve historical retention
    1. Results


Notification #2: read
Notification #3: unread
Notification #4: read


  1. 9:15pm – Historical retention check is run
    1. Notification #2 removed from retention since it is more than 12 hours old.
  2. 10:20pm – Historical retention check is run
    1. Notification #3 removed from retention since it is more than 12 hours old.
  3. 10:30pm – Historical retention check is run
    1. Notification #4 removed from retention since it is more than 12 hours old.
  4. 10:45pm API call to retrieve historical retention
    1. Results – Empty list
    2. All notification have been removed.


Search