Working with the Events Store

The /webhooks/subscriptions/{subscriptionId}/events endpoint returns information about the events that have been generated for a given webhooks subscription. For example:

{
   "id": "28da8068-61cc-4af3-8d5e-ae49a02dd47b",
   "createdAt": "2020-01-27T16:53:50.815936Z",
   "updatedAt": "2020-01-27T16:53:51.554764Z",
   "state": "success",
   "attempts": 1,
    "request": {
       "endpoint": "https://webhook.site/46ff3c5e-ae95-43df-b32d-d07bb84746b4",
      "headers": {
           "Accept": "*/*",
           "Content-Length": "1293",
           "Content-Type": "application/secevent+jwt",
           "Host": "webhook.site",
           "User-Agent": "Akamai Identity Cloud Webhooks/v3.0.0"
       },
       "payload": "eyJhbGciOiJSUzI1NiIsImtpZCI6IjFkYzEyMDczNjk5YzY4YzFkYWVlNmM5YTEwMGUyYjQzZmViZGNkOTIifQ
.eyJhdWQiOlsiaHR0cHM6Ly93ZWJob29rLnNpdGUvNDZmZjNjNWUtYWU5NS00M2RmLWIzMmQtZDA3Y
mI4NDc0NmI0Il0sImV2ZW50cyI6eyJlbnRpdHlVcGRhdGVkIjp7ImF0dHJpYnV0ZXMiOlsiZmFtaWx5TmFtZSJ
dLCJjYXB0dXJlQXBwbGljYXRpb25JZCI6IjN2YWRiYTN2aHFwa2RndHNycWQ0c3Q3Nm0zIiwiY2FwdHVyZ
UNsaWVudElkIjoiM2ZwNHp0OXQyNTZqcWs0dHgzNXd1ajRhcDJlNTNocTkiLCJlbnRpdHlUeXBlIjoidXNlciIs
Imdsb2JhbFN1YiI6ImNhcHR1cmUtdjE6Ly9hcHAuY2FwdHVyZS5tdWx0aS5kZXYub3IuamFucmFpbi5jb20v
M3ZhZGJhM3ZocXBrZGd0c3JxZDRzdDc2bTMvdXNlci9lNDk5YWMyNC00NmJkLTRkODUtOTFlZS1iZGVhZ
GQ0NDZjMWUiLCJzdWIiOiJlNDk5YWMyNC00NmJkLTRkODUtOTFlZS1iZGVhZGQ0NDZjMWUifX0sImlhdC
I6MTU4MDE0NDAzMCwiaXNzIjoiaHR0cHM6Ly9hcGkubXVsdGkuZGV2Lm9yLmphbnJhaW4uY29tLzAwMD
AwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMC93ZWJob29rcyIsImp0aSI6IjFlNTY5Y
mI1LWMyMWQtNGU5Yy1hMWFhLTdlYzNiODMxYmRhMiIsInRvZSI6MTU4MDE0NDAzMDczOSwidHhuIjoiM
DAwMDAwMDAtMDAwMC0wMDAwLTAwMDAtMDAwMDAwMDAwMDAwIn0.D4PsqQiwvXmd3bons9Jcq
HEf69Tw46I2C4Zg713gw8Y11QPLywKXMveFzgmRRVS3OyG38LVSz6wJfmn02Du4hlMpi44Rnna_k9rv1W
TxC7CFOFqVXzJyQ-TvSdYLEU5hfCpfodlyeE4v-SPZoOfZkQdUa8vPm4wiOOnr0QfZlY3uTRcPczY3jy2Gv6eL
RSjk2q4LY0rYHHaojHofYJARahGjah9JrkmbTaMkvuv3CwBnUrKkp2nomOzl2Z858xaRUrgHfgXEuOYrhasRrbc
An-MyuQfY7hlPjSl8Lka25JgeIOmCLpK2-fFZ0rDW1k1MzHak6q2rftE7vSvObcw"
   },
   "response": {
       "statusCode": 200,
       "headers": {
           "Cache-Control": "no-cache, private",
           "Content-Type": "text/plain; charset=UTF-8",
           "Date": "Mon, 27 Jan 2020 16:53:51 GMT",
           "Server": "nginx/1.14.2",
           "Set-Cookie": "laravel_session=0WvS2SrKRs0In3XSBAdlLykpCCTTnJmzLnCYljfw; expires=Mon, 27-Jan-2020 18:53:51 GMT;    Max-Age=7200; path=/; httponly",
           "Vary": "Accept-Encoding",
           "X-Ratelimit-Limit": "100",
           "X-Ratelimit-Remaining": "97",
           "X-Request-Id": "11d339a3-17bd-457d-abde-64c5810e5244",
           "X-Token-Id": "46ff3c5e-ae95-43df-b32d-d07bb84746b4"
       }
   },
   "_links": {
       "self": {
           "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/e5269842-024f-49bf-8a39-67aec8048b42/events/28da8068-61cc-4af3-8d5e-ae49a02dd47b"
       },
       "redeliver": {
           "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/e5269842-024f-49bf-8a39-67aec8048b42/events/28da8068-61cc-4af3-8d5e-ae49a02dd47b/redeliver"
       },
       "history": {
           "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/e5269842-024f-49bf-8a39-67aec8048b42/events/28da8068-61cc-4af3-8d5e-ae49a02dd47b/history"
       }
   },
   "eventType": "entityUpdated"
}

To be honest, the actual data returned here doesn’t matter; what matters is that the /subscriptions/{subscriptionId}/events endpoints are connected to the Identity Cloud events store; they are not connected to the database of received webhook notifications maintained by your organization. Among other things, that explains why the payload value for each event looks similar to this:

Y4YzFkYWVlNmM5YTEwMGUyYjQzZmViZGNkOTIifQ.eyJpc3MiOiJBa2FtYWkgSWRlbnRpdHkgQ2xvdWQ
iLCJpYXQiOjE1NjM0ODg2MzEsImp0aSI6ImI3MDA0NmJkLTQ0YzctNDU3NS1iMWEyLTliODU1NmQxZjA0M
CIsImF1ZCI6Imh0dHBzOi8vZXhhbXBsZS5jb20vcGF0aC90by9lbmRwb2ludCIsInR4biI6IjAwMDAwMDAwL
TAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCIsInRvZSI6MTU1OTM3MjQwMCwiZXZlbnRzIjp
7InVzZXJDcmVkZW50aWFsVXBkYXRlZCI6eyJjYXB0dXJlQXBwbGljYXRpb25JZCI6Inp6eW45Z3k5cjh4ZHk
1emtydTR5NTRzeWs2IiwiY2FwdHVyZUNsaWVudElkIjoiZWxycm5pdXg1MWEzbnJoZnd6a2x2ejN0NDZsY
jVuMm0iLCJlbnRpdHlUeXBlIjoidXNlciIsImdsb2JhbFN1YiI6ImNhcHR1cmUtdjE6Ly91cy5qYW5yYWluY2Fwd
HVyZS5jb20venp5bjlneTlyOHhkeTV6a3J1NHk1NHN5azYvdXNlci82YjAwNGJjNS0xNzljLTQ1YzItODE1ZC0z
MWIwNjE2OTM3MWQiLCJzdWIiOiI2YjAwNGJjNS0xNzljLTQ1YzItODE1ZC0zMWIwNjE2OTM3MWQiLCJjcm
VkZW50aWFsVHlwZSI6InBhc3N3b3JkIiwiaWQiOiIwMDAwMDAwMC0wMDAwLTAwMDAtMDAwMC0wM
DAwMDAwMDAwMDAifX19.IvkrGFE3GsK3eTLO_QvdFKqg4ktJ2sDToHNghMfGUlWNzRLMIpmgsWZXzLv0
QMiyatLva7mEshTlfyOje-S_Y-nUniM9hgHgNg-R0Az9hs2mu_ORcXEFo9AHayhjvW1bKHcmTI7dlw2fqFl-2V
BS594LQspDYfZ4WJ7hexq7OwACB8qp0oVskx_fc8mHQfy4YnW5GF4XlTcl6CnjYU81qY4hejcnkkg8olbq_eP
UnpTpW8-YO5cPW9nW8KlivRJGWJbEXnffSAd5

That’s because security event tokens are not decoded before being added to the events store. Instead, events are written to the database as-is.

Does it really matter that the APIs are connected to the event store rather than to your organization’s internal database? As a matter of fact, it does. Your organization’s internal database shows the event notifications that have been delivered. By comparison, the event store shows every event that has been generated for your organization. (Or, to be a little more precise, it shows every event that has been generated in the past 7 days; events more than 7 days old are automatically deleted from the events store.)

So why does that matter? That matters because it gives you a way to check how well the webhooks delivery service is working. If you thumb through the event store and see one or two failed deliveries (among thousands of events), that’s probably no cause for alarm. If you see hundreds of failed events, that could be a different story.

Equally important, this also provides a way for you to reschedule delivery for events that, for whatever reason, couldn’t be delivered. In Webhooks v2, the system tries 3 times to deliver an event; if delivery can’t be made after 3 tries that event is deleted and is no longer be accessible.

In Webhooks v3, the system makes 6 attempts to deliver an event. And what if delivery fails all 6 times? Well, in that case, the event state changes to failure, and no more delivery attempts are made. However, the event is not discarded, at least not right away. Instead, failed events are maintained in the events store for 7 days; at any time during those 7 days you can schedule the events for a redelivery attempt.  And how do you know if you even have any failed events? That’s right: you can use the /subscriptions/{subscriptionId}/events endpoint to query the event store, then check for events where the state is set to failure.

Tip. Curious as to why an event failed? To help troubleshoot delivery issues, use the /{customerId}/webhooks/subscriptions/{subscriptionId}/events/{eventId}/history endpoint to trace the complete delivery (or lack of delivery) history for a given event. For example, this Curl command traces the delivery history for the d375d2f8-e2d8-4859-9c31-648468b80acd:

curl -X GET \
 
https://v1.api.us.janrain.com/9bc867ed-1f10-420f-8d90-398fde4e4779/webhooks/subscriptions/454fe969-1909-4e93-b552-674d47eafdb0/events/d375d2f8-e2d8-4859-9c31-648468b80acd/history \
 -H 'Authorization: Bearer Xk7EzdpGq5GPQcsxCWM2SxdlwU_iTsA4i2Px4TEzBrfLIvddjnDVBJxjPDuCARHH'

It’s useful to know that an event (or a set of events) could not be delivered; it’s equally useful to be able to use the /history endpoint to try and figure out why an event (or set of events) couldn’t be delivered. However, knowing that an event wasn’t delivered, and even knowing why an event wasn’t delivered, still leaves us with one problem: the event wasn’t delivered. Yes, the event remains in the event store for 7 days, but after 7 days it disappears for good. And that’s a problem: after all, the whole idea behind webhooks is to make sure that, whenever events of interest occur, you get your very own copy of those events. 

But before you start feeling too sorry for yourself, here’s a suggestion: if you have one or more delivery failures, just schedule those events for redelivery. When you schedule an event for redelivery, its event state gets changed from failure to awaiting-retry. That’s a signal that this event needs to go back through the delivery cycle. Sooner or later, the system will attempt to redeliver each awaiting-retry event., If that doesn’t work, 5 more delivery attempts will be made over the course of the next 24 hours. Only then, after 6 failed delivery attempts, will the event be marked as failure. You’ll then have 7 days in which to schedule another redelivery attempt. 

When it comes to event redeliveries you have two options: you can reschedule all your failed events for delivery, or you can schedule individual events for delivery. To redeliver all your failed events, use the /webhooks/subscriptions/{subscriptionId}/events/redeliver API endpoint and a Curl command similar to this:

curl -X POST \
  https://v1.api.us.janrain.com/9bc867ed-1f10-420f-8d90-398fde4e4779/webhooks/subscriptions/454fe969-1909-4e93-b552-674d47eafdb0/events/redeliver \
  -H 'Authorization: Bearer Xk7EzdpGq5GPQcsxCWM2SxdlwU_iTsA4i2Px4TEzBrfLIvddjnDVBJxjPDuCARHH' 

Alternatively, you can schedule individual events for redelivery. For example, this command configures event d375d2f8-e2d8-4859-9c31-648468b80acd for redelivery:

curl -X POST \
  https://v1.api.us.janrain.com/9bc867ed-1f10-420f-8d90-398fde4e4779/webhooks/subscriptions/454fe969-1909-4e93-b552-674d47eafdb0/events/d375d2f8-e2d8-4859-9c31-648468b80acd/redeliver \
  -H 'Authorization: Bearer Xk7EzdpGq5GPQcsxCWM2SxdlwU_iTsA4i2Px4TEzBrfLIvddjnDVBJxjPDuCARHH'