Working with the Events Store

The /webhooks/subscriptions/{subscriptionId}/events endpoint returns information about the events that have been generated for a given webhooks subscription. For example:

{
   "id": "28da8068-61cc-4af3-8d5e-ae49a02dd47b",
   "createdAt": "2020-01-27T16:53:50.815936Z",
   "updatedAt": "2020-01-27T16:53:51.554764Z",
   "state": "success",
   "attempts": 1,
    "request": {
       "endpoint": "https://webhook.site/46ff3c5e-ae95-43df-b32d-d07bb84746b4",
      "headers": {
           "Accept": "*/*",
           "Content-Length": "1293",
           "Content-Type": "application/secevent+jwt",
           "Host": "webhook.site",
           "User-Agent": "Akamai Identity Cloud Webhooks/v3.0.0"
       },
       "payload": "eyJhbGciOiJSUzI1NiIsImtpZCI6IjFkYzEyMDczNjk5YzY4YzFkYWVlNmM5YTEwMGUyYjQzZmViZGNkOTIifQ.eyJhdWQiOlsiaHR0cHM6Ly93ZWJob29rLnNpdGUvNDZmZjNjNWUtYWU5NS00M2RmLWIzMmQtZDA3YmI4NDc0NmI0Il0sImV2ZW50cyI6eyJlbnRpdHlVcGRhdGVkIjp7ImF0dHJpYnV0ZXMiOlsiZmFtaWx5TmFtZSJdLCJjYXB0dXJlQXBwbGljYXRpb25JZCI6IjN2YWRiYTN2aHFwa2RndHNycWQ0c3Q3Nm0zIiwiY2FwdHVyZUNsaWVudElkIjoiM2ZwNHp0OXQyNTZqcWs0dHgzNXd1ajRhcDJlNTNocTkiLCJlbnRpdHlUeXBlIjoidXNlciIsImdsb2JhbFN1YiI6ImNhcHR1cmUtdjE6Ly9hcHAuY2FwdHVyZS5tdWx0aS5kZXYub3IuamFucmFpbi5jb20vM3ZhZGJhM3ZocXBrZGd0c3JxZDRzdDc2bTMvdXNlci9lNDk5YWMyNC00NmJkLTRkODUtOTFlZS1iZGVhZGQ0NDZjMWUiLCJzdWIiOiJlNDk5YWMyNC00NmJkLTRkODUtOTFlZS1iZGVhZGQ0NDZjMWUifX0sImlhdCI6MTU4MDE0NDAzMCwiaXNzIjoiaHR0cHM6Ly9hcGkubXVsdGkuZGV2Lm9yLmphbnJhaW4uY29tLzAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMC93ZWJob29rcyIsImp0aSI6IjFlNTY5YmI1LWMyMWQtNGU5Yy1hMWFhLTdlYzNiODMxYmRhMiIsInRvZSI6MTU4MDE0NDAzMDczOSwidHhuIjoiMDAwMDAwMDAtMDAwMC0wMDAwLTAwMDAtMDAwMDAwMDAwMDAwIn0.D4PsqQiwvXmd-3bons9JcqHEf69Tw46I2C4Zg713gw8Y11QPLywKXMveFzgmRRVS3OyG38LVSz6wJfmn02Du4hlMpi44Rnna_k9rv1WTxC7CFOFqVXzJyQ-TvSdYLEU5hfCpfodlyeE4v-SPZoOfZ-kQdUa8vPm4wiOOnr0QfZlY3uTRcPczY3jy2Gv6eLRSjk2q4LY0-rYHHaojHofYJARahGjah9JrkmbTaMkvuv3CwBnUrKkp2nomOzl2Z858xaRUrgHfgXEuOYrhasRrbcAn-MyuQfY7hlPjSl8Lka25JgeIOmCLpK2-fFZ0rDW1k1MzHak6q2rftE7vSvObcw"
   },
   "response": {
       "statusCode": 200,
       "headers": {
           "Cache-Control": "no-cache, private",
           "Content-Type": "text/plain; charset=UTF-8",
           "Date": "Mon, 27 Jan 2020 16:53:51 GMT",
           "Server": "nginx/1.14.2",
           "Set-Cookie": "laravel_session=0WvS2SrKRs0In3XSBAdlLykpCCTTnJmzLnCYljfw; expires=Mon, 27-Jan-2020 18:53:51 GMT;    Max-Age=7200; path=/; httponly",
           "Vary": "Accept-Encoding",
           "X-Ratelimit-Limit": "100",
           "X-Ratelimit-Remaining": "97",
           "X-Request-Id": "11d339a3-17bd-457d-abde-64c5810e5244",
           "X-Token-Id": "46ff3c5e-ae95-43df-b32d-d07bb84746b4"
       }
   },
   "_links": {
       "self": {
           "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/e5269842-024f-49bf-8a39-67aec8048b42/events/28da8068-61cc-4af3-8d5e-ae49a02dd47b"
       },
       "redeliver": {
           "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/e5269842-024f-49bf-8a39-67aec8048b42/events/28da8068-61cc-4af3-8d5e-ae49a02dd47b/redeliver"
       },
       "history": {
           "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/e5269842-024f-49bf-8a39-67aec8048b42/events/28da8068-61cc-4af3-8d5e-ae49a02dd47b/history"
       }
   },
   "eventType": "entityUpdated"
}

To be honest, the actual data returned here doesn’t matter; what matters is that the /subscriptions/{subscriptionId}/events endpoints are connected to the Identity Cloud events store; they are not connected to the database of received webhook notifications maintained by your organization. Among other things, that explains why the payload value for each event looks similar to this:

Y4YzFkYWVlNmM5YTEwMGUyYjQzZmViZGNkOTIifQ.eyJpc3MiOiJBa2FtYWkgSWRlbnRpdHkgQ2xvdWQiLCJpYXQiOjE1NjM0ODg2MzEsImp0aSI6ImI3MDA0NmJkLTQ0YzctNDU3NS1iMWEyLTliODU1NmQxZjA0MCIsImF1ZCI6Imh0dHBzOi8vZXhhbXBsZS5jb20vcGF0aC90by9lbmRwb2ludCIsInR4biI6IjAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCIsInRvZSI6MTU1OTM3MjQwMCwiZXZlbnRzIjp7InVzZXJDcmVkZW50aWFsVXBkYXRlZCI6eyJjYXB0dXJlQXBwbGljYXRpb25JZCI6Inp6eW45Z3k5cjh4ZHk1emtydTR5NTRzeWs2IiwiY2FwdHVyZUNsaWVudElkIjoiZWxycm5pdXg1MWEzbnJoZnd6a2x2ejN0NDZsYjVuMm0iLCJlbnRpdHlUeXBlIjoidXNlciIsImdsb2JhbFN1YiI6ImNhcHR1cmUtdjE6Ly91cy5qYW5yYWluY2FwdHVyZS5jb20venp5bjlneTlyOHhkeTV6a3J1NHk1NHN5azYvdXNlci82YjAwNGJjNS0xNzljLTQ1YzItODE1ZC0zMWIwNjE2OTM3MWQiLCJzdWIiOiI2YjAwNGJjNS0xNzljLTQ1YzItODE1ZC0zMWIwNjE2OTM3MWQiLCJjcmVkZW50aWFsVHlwZSI6InBhc3N3b3JkIiwiaWQiOiIwMDAwMDAwMC0wMDAwLTAwMDAtMDAwMC0wMDAwMDAwMDAwMDAifX19.IvkrGFE3GsK3eTLO_QvdFKqg4ktJ2sDToHNghMfGUlWNzRLMIpmgsWZXzLv0QMiyatLva7mEshTlfyOje-S_Y-nUniM9hgHgNg-R0Az9hs2mu_ORcXEFo9AHayhjvW1bKHcmTI7dlw2fqFl-2VBS594LQspDYfZ4WJ7hexq7OwACB8qp0oVskx_fc8mHQfy4YnW5GF4XlTcl6CnjYU81qY4hejcnkkg8olbq_ePUnpTpW8-YO5cPW9nW8KlivRJGWJbEXnffSAd5

That’s because security event tokens are not decoded before being added to the events store. Instead, events are written to the database as-is.

Does it really matter that the APIs are connected to the event store rather than to your organization’s internal database? As a matter of fact, it does. Your organization’s internal database shows the event notifications that have been delivered. By comparison, the event store shows every event that has been generated for your organization. (Or, to be a little more precise, it shows every event that has been generated in the past 7 days; events more than 7 days old are automatically deleted from the events store.)

So why does that matter? That matters because it gives you a way to check how well the webhooks delivery service is working. If you thumb through the event store and see one or two failed deliveries (among thousands of events), that’s probably no cause for alarm. If you see hundreds of failed events, that could be a different story.

Equally important, this also provides a way for you to reschedule delivery for events that, for whatever reason, couldn’t be delivered. In Webhooks v2, the system tries 3 times to deliver an event; if delivery can’t be made after 3 tries that event is deleted and is no longer be accessible.

In Webhooks v3, the system makes 6 attempts to deliver an event. And what if delivery fails all 6 times? Well, in that case, the event state changes to failure, and no more delivery attempts are made. However, the event is not discarded, at least not right away. Instead, failed events are maintained in the events store for 7 days; at any time during those 7 days you can schedule the events for a redelivery attempt.  And how do you know if you even have any failed events? That’s right: you can use the /subscriptions/{subscriptionId}/events endpoint to query the event store, then check for events where the state is set to failure.

Tip. Curious as to why an event failed? To help troubleshoot delivery issues, use the /{customerId}/webhooks/subscriptions/{subscriptionId}/events/{eventId}/history endpoint to trace the complete delivery (or lack of delivery) history for a given event. For example, this Curl command traces the delivery history for the d375d2f8-e2d8-4859-9c31-648468b80acd:

curl -X GET \
 
https://v1.api.us.janrain.com/9bc867ed-1f10-420f-8d90-398fde4e4779/webhooks/subscriptions/454fe969-1909-4e93-b552-674d47eafdb0/events/d375d2f8-e2d8-4859-9c31-648468b80acd/history \
 -H 'Authorization: Bearer Xk7EzdpGq5GPQcsxCWM2SxdlwU_iTsA4i2Px4TEzBrfLIvddjnDVBJxjPDuCARHH'

It’s useful to know that an event (or a set of events) could not be delivered; it’s equally useful to be able to use the /history endpoint to try and figure out why an event (or set of events) couldn’t be delivered. However, knowing that an event wasn’t delivered, and even knowing why an event wasn’t delivered, still leaves us with one problem: the event wasn’t delivered. Yes, the event remains in the event store for 7 days, but after 7 days it disappears for good. And that’s a problem: after all, the whole idea behind webhooks is to make sure that, whenever events of interest occur, you get your very own copy of those events. 

But before you start feeling too sorry for yourself, here’s a suggestion: if you have one or more delivery failures, just schedule those events for redelivery. When you schedule an event for redelivery, its event state gets changed from failure to awaiting-retry. That’s a signal that this event needs to go back through the delivery cycle. Sooner or later, the system will attempt to redeliver each awaiting-retry event., If that doesn’t work, 5 more delivery attempts will be made over the course of the next 24 hours. Only then, after 6 failed delivery attempts, will the event be marked as failure. You’ll then have 7 days in which to schedule another redelivery attempt. 

When it comes to event redeliveries you have two options: you can reschedule all your failed events for delivery, or you can schedule individual events for delivery. To redeliver all your failed events, use the /webhooks/subscriptions/{subscriptionId}/events/redeliver API endpoint and a Curl command similar to this:

curl -X POST \
  https://v1.api.us.janrain.com/9bc867ed-1f10-420f-8d90-398fde4e4779/webhooks/subscriptions/454fe969-1909-4e93-b552-674d47eafdb0/events/redeliver \
  -H 'Authorization: Bearer Xk7EzdpGq5GPQcsxCWM2SxdlwU_iTsA4i2Px4TEzBrfLIvddjnDVBJxjPDuCARHH' 

Alternatively, you can schedule individual events for redelivery. For example, this command configures event d375d2f8-e2d8-4859-9c31-648468b80acd for redelivery:

curl -X POST \
  https://v1.api.us.janrain.com/9bc867ed-1f10-420f-8d90-398fde4e4779/webhooks/subscriptions/454fe969-1909-4e93-b552-674d47eafdb0/events/d375d2f8-e2d8-4859-9c31-648468b80acd/redeliver \
  -H 'Authorization: Bearer Xk7EzdpGq5GPQcsxCWM2SxdlwU_iTsA4i2Px4TEzBrfLIvddjnDVBJxjPDuCARHH'