Why Can't I Find the User Account I Just Created?

Everyone loses things from time-to-time: car keys, cell phones, TV remotes, whatever. And that’s OK: these things happen. But what are you supposed to do when you “lose” one of your Identity Cloud user accounts?

Now, before we go any further, we should note two things. First, people who have run into the situation we’re about to discuss haven’t really lost any user accounts; instead they created a new account and then immediately tried – and failed – to find that account. But the account wasn’t lost: it just took a few minutes before the entity.find endpoint was able to discover it. 


Second, you might be thinking, “I’ve created tons of user accounts and I’ve never ‘lost’ any of them.” Fortunately, you’re far from alone: this situation pops up only on very rare occasions, and there’s a good chance that you’ll never encounter it. But it can happen, which is the reason for this article: we’re going to explain why you might create a new user record and then be unable to immediately access that record using either the entity.find or the entity.count endpoints. And, as a bonus, we’ll explain what you can do to help workaround this event should you ever run into it.



What Do We Mean When We Say We Can’t Find the User Account We Just Created?


To answer that question, let’s walk through a situation that could occur, and then explain why the issue might crop up (and what you can do about it).


We’ll start with a sample Identity Cloud implementation that has 10 users, with all those user records stored in the user profile store. We know that we have 10 users because that’s what comes back when we call the entity.count endpoint:


{
    "total_count": 10,
    "stat": "ok"
}


Now we’ll use the entity.create endpoint to create a new user: Karim Nafir. Here’s the API call that creates the user account:


curl -X POST \
  -H "Authorization: Basic odw7Zmc2ZjQ0bXNhYzN2ZXBqanZ4Z2d6dnQzZTNzazk6OTVjY3hrN2HB8up1eng2ZHB0ZTV
rOXA2ZGo1KJ7fge3=" \
  --data-urlencode type_name=user \
  --data-urlencode attributes='{"givenName":"Karim","familyName":"Nafir","email":"karim.nafir@mail.com", 
"displayName":"Karim.Nafir","gender":"male","birthday":"12/19/1989"}' \
  https://se-demos-gstemp.us-dev.janraincapture.com/entity.create


Did our API call succeed? Yes it did; in fact, we got back an API response that includes the UUID and the ID of our new user:


{
    "uuid": "73887920-936f-4af9-9933-32eb036d68b6",
    "stat": "ok",
    "id": 8065
}


So far so good.


But here’s where things can get a little tricky. (Although, at the risk of repeating ourselves, we should emphasize that this might never happen to you; to a large extent, this as much a matter of bad luck as it is anything else.) Suppose we create our new account and then immediately call the entity.count endpoint. Here’s what we get back:


{
    "total_count": 10,
    "stat": "ok"
}


Needless to say, that’s not what we expected to get back. After all, we already had 10 users, and then we created a new user account. That means we should have 11 users: 10 existing users plus 1 new one. And yet we don’t.


As a sanity check, we use entity.find to search for our newly-created user account, employing the UUID returned from the entity.create call as our unique identifier. Here’s that API call:


curl -X POST 'https://se-demos-gstemp.us-dev.janraincapture.com/entity.find' \
  --data-urlencode type_name=user \
  --data-urlencode filter="uuid='73887920-936f-4af9-9933-32eb036d68b6'" \
-H 'Authorization: Basic eTR4Zmc2ZjQ0bXNhYzN2ZXBqanZ4Z2d6dnQzZTNzazk6OTVjY3hrN2N6YnZ1eng2ZHB0ZTVrOX
A2ZGo1Ynpla3U='


And here’s the response we get back:


{
    "results": [],
    "result_count": 0,
    "stat": "ok"
}


If this happens to you it’s reasonable to assume that the account never got created: after all, it doesn’t show up when you count the records and it doesn’t show up when you search for the record. Based on that, you'll probably call entity.create and take a second crack at creating the record. So what happens when you do that? This happens:


{
    "attribute_name": "/email",
    "error": "unique_violation",
    "request_id": "jvhmramp2er3nfer",
    "error_description": "Attempted to update a duplicate value: attribute '/email' to value 
karim.nafir@mail.com'",
    "stat": "error",
    "code": 361
}


According to the error message, you already have a record for the user with the email address karim.nafir@mail.com. And because email addresses must be unique, your API call fails.


In other words, entity.create insists you created the record, but entity.find and entity.count are equally insistent that you didn’t create the record:



It’s a paradox worth of Edwin Schrodinger: your new user record apparently both exists and doesn’t exist at the same time!



So What’s Going On Here?


Understanding Schrodinger’s cat paradox requires you to consider everything from quantum superpositions to the many-world theory to objective collapse postulates; fortunately, our Identity Cloud user record paradox is much easier to explain. As it turns out, any time you create a new user record (regardless of whether you use the entity.create endpoint or the /oauth/register_native_traditional endpoint or any other mechanism that can create user records) that account is immediately written to the Identity Cloud’s primary datastore. And that’s a good thing: if a user clicks a Create Account button you want the new account to be created as quickly as possible. (And, by extension, to be available for use as quickly as possible.)


In other words, when you call entity.create a new record is created in the blink of an eye. So then why can’t you actually find that record? To explain why this is the case, let’s first note that creating new records doesn’t put much stress on a server or on the network; for example, in our sample API call all we did was set the values of a half-dozen or so attributes:


--data-urlencode attributes='{"givenName":"Karim","familyName":"Nafir","email":"karim.nafir@mail.com", 
"displayName":"Karim.Nafir","gender":"male","birthday":"12/19/1989"}' \


Not a big deal at all.


However, this isn’t necessarily the case when it comes to searching for, or counting, all your user records. For example, suppose Organization A has 10 million records, and it now wants to return profile information for all the male users under 30 years of age who live in one of a specified set of European countries and who haven’t logged on in the past 90 days. As you might expect, that’s a much more resource-intensive operation. And if Organizations B and C happen to be conducting similar searches at the same time, the potential exists to considerably slow down the system for all Identity Cloud customers.


Or at least it would if entity.find and entity.count worked against the primary datastore. But that’s not the way the system works. Instead, when a new user record is created that record is written to the primary datastore. When that’s done, a copy of that record is added to a read-only replica of the primary datastore. When you call entity.find or entity.count, those endpoints work against the replica datastore; neither entity.find nor entity.count ever touches the primary datastore. By moving resource-intensive procedures to the replica datastore, the Identity Cloud can make user records readily available for searching and, at the same time, save wear-and-tear on the primary datastore. 


In admittedly-simplistic fashion, the process looks like this:



As a general rule, replication is extremely fast: the process rarely takes more than a few milliseconds to complete. Because of that, the odds are pretty good that you’ll never know that entity.find or entity.count are working against a replica datastore: if you use entity.create to create a new record and then immediately call entity.find, you’ll probably find your record. That’s how fast replication usually works.


But “usually” isn’t synonymous with “always.” There are factors – server workload, network congestion, hardware problems – that can slow replication. These circumstances are rare, and “slowing” replication typically means that the process takes one full second rather than a few milliseconds. But, rare as it might be, it is possible for you to create a record and then not have instant access to that record by using entity.find or entity.count. If that happens it's invariably due to a replication delay: entity.find simply started searching the replica database before the new record was copied over. As a result, there was nothing to find.


In other words:


  1. When a new user record is created, that record is created in the primary datastore.
  2. The new record is then copied to a replica datastore.
  3. When you call the entity.find or entity.count endpoints, those endpoints query the replica datastore.
  4. If, for some reason, the replicated record hasn’t arrived, then the record won’t be in the replica datastore. As a result, neither entity.find nor entity.count will have any knowledge of it.

Or:



Incidentally, you can run into this same problem if you update or delete a user profile: that’s because updates and deletes also work against the primary datastore, and any changes you make need to be copied to the replica datastore. If you delete a record right this very moment, it’s possible that, a few seconds from now, that record will still be in the replica datastore. Again, highly unlikely, but possible.


Oh: and you can also encounter a delay when viewing user accounts in Console. That’s because Console also uses a replica datastore when searching for user records. 



How Can I Work Around This Issue?


To be honest, that depends on what you think the issue is. If you think the issue is, “How can I directly search the primary datastore in order to avoid potential information lags?” then the answer is pretty straightforward: you can’t. The entity.find and entity.create endpoints always work against the replica datastore. Meanwhile the entity endpoints for creating, updating, and deleting records always work against the primary datastore:

On the other hand, suppose you think the issue is this: “How can I work with a new record immediately after it’s been created?” (For example, maybe you want to grab user profile information in order to provide the user with a personalized website or a custom newsfeed.) In that case, we have two recommendations: 


  1. Use the include_record parameter when making your call to entity.create.
  2. Use the entity endpoint (and not entity.find) if you need to retrieve the record.


As we’ve already seen, calling entity.create without using the include_record parameter returns a minimal amount of information for the new record:


{
    "uuid": "73887920-936f-4af9-9933-32eb036d68b6",
    "stat": "ok",
    "id": 8065
}


So let’s add the include_record parameter to our API call and then compare that API response to the preceding response. First, take a peek at a sample curl command that sets include_record to true:


curl -X POST \
  -H "Authorization: Basic odw7Zmc2ZjQ0bXNhYzN2ZXBqanZ4Z2d6dnQzZTNzazk6OTVjY3hrN2HB8up1eng2ZHB0ZTV
rOXA2ZGo1KJ7fge3" \
  --data-urlencode type_name=user \
  --data-urlencode include_record=true \
  --data-urlencode attributes='{"givenName":"Karim","familyName":"Nafir","email":"karim.nafir@mail.com", 
"displayName":"Karim.Nafir","gender":"male","birthday":"12/19/1989"}' \
  https://se-demos-gstemp.us-dev.janraincapture.com/entity.create


And here’s the API response we get this time:


{
    "uuid": "f719051b-e109-4618-815f-0468bbc971ce",
    "result": {
        "givenName": "Karim",
        "fullName": null,
        "email": "karim.nafir@mail.com",
        "accountDeleteRequestTime": null,
        "clientId": null,
        "lastUpdated": "2020-08-19 16:29:08.593749 +0000",
        "roles": [],
        "display": null,
        "middleName": null,
        "primaryAddress": {
            "address2": null,
            "phone": null,
            "country": null,
            "zip": null,
            "company": null,
            "city": null,
            "address1": null,
            "zipPlus4": null,
            "stateAbbreviation": null
        },
        "accountDataRequestTime": null,
        "consents": {
            "marketing": {
                "clientId": null,
                "granted": null,
                "context": null,
                "updated": null,
                "type": null
            }
        },
        "uuid": "f719051b-e109-4618-815f-0468bbc971ce",
        "created": "2020-08-19 16:29:08.593749 +0000",
        "profiles": [],
        "mobileNumber": null,
        "deactivateAccount": null,
        "birthday": "1989-12-19",
        "lastLogin": null,
        "familyName": "Nafir",
        "gender": "male",
        "photos": [],
        "password": null,
        "clients": [],
        "emailVerified": null,
        "displayName": "Karim Nafir",
        "id": 8065,
        "mobileNumberVerified": null,
        "externalId": null,
        "legalAcceptances": []
    },
    "stat": "ok",
    "id": 8065
}


Basically, this is the user profile, including:


  • All the values we get back if we don’t use include_record (shown in blue).
  • All the values we set in our call to entity.create (shown in red),.
  • All the remaining values/attributes found in the user profile. We don’t have any values for these attributes (other than null) simply because we didn’t supply any values when we created the record.

This is as up-to-date a user profile as you can get.


Another option is to use the entity endpoint instead of the entity.find. If you want to retrieve information for just one user record (as opposed to multiple records), the entity endpoint is the recommended way to go. And that’s true even if you aren’t running into replication delays. There are at least two reasons for that: 


  1. The entity endpoint runs against the primary datastore, which means you’re always working with the latest version of the user profile.

  2. Performance tends to be faster and more efficient  because the entity endpoint goes directly to the specified record, without having to search the entire datastore. Not coincidentally, this also results in less wear and tear being placed on the server.

As long as you have a unique identifier for the user (such as the user’s id or uuid, or the user’s emailaddress) then retrieving a user record by calling the entity endpoint is easy. For example, the following curl command returns the user profile for the user with the UUID 72ef339b-192c-47bc-95a2-a395be84b57c:


curl -X POST 'https://se-demos-gstemp.us-dev.janraincapture.com/entity' \
  -H 'Authorization: Basic odw7Zmc2ZjQ0bXNhYzN2ZXBqanZ4Z2d6dnQzZTNzazk6OTVjY3hrN2HB8up1eng2ZHB0ZTV
rOXA2ZGo1KJ7fge3' \
  --data-urlencode type_name=USER \
  --data-urlencode uuid=72ef339b-192c-47bc-95a2-a395be84b57c


And here’s the kind of data (retrieved from the primary data store) you can expect to get back:


{
    "result": {
        "givenName": "August",
        "fullName": null,
        "email": "augustjosephspringer@gmail.com",
        "accountDeleteRequestTime": null,
        "clientId": null,
        "lastUpdated": "2020-08-26 14:12:10.550244 +0000",
        "roles": [],
        "display": null,
        "middleName": "Joseph",
        "primaryAddress": {
            "address2": "Suite 150",
            "phone": "+1-8773252624",
            "country": "US",
            "zip": "97209",
            "company": "Akamai",
            "city": "Portland",
            "address1": "1233 NW 12th Ave",
            "zipPlus4": null,
            "stateAbbreviation": "OR"
        },
        "accountDataRequestTime": null,
        "consents": {
            "marketing": {
                "clientId": null,
                "granted": null,
                "context": null,
                "updated": null,
                "type": null
            }
        },
        "uuid": "72ef339b-192c-47bc-95a2-a395be84b57c",
        "created": "2020-04-15 21:57:46.405813 +0000",
        "profiles": [],
        "mobileNumber": "1-425-555-0903",
        "deactivateAccount": null,
        "birthday": "1989-12-19",
        "lastLogin": "2020-08-24 18:04:29 +0000",
        "familyName": "Springer",
        "gender": "male",
        "photos": [],
        "password": {
            "value": "$2a$10$3YkOFhnTjVYRgtgegOqo3.HMbnZ6PDjFMIukgfAIkbnS8WtDPdkh2",
            "type": "password-bcrypt"
        },
        "clients": [
            {
                "clientId": "u74hp2xa4u75dq9s6wv8yyb28wkkux7m",
                "lastLogin": "2020-06-01 18:02:46 +0000",
                "name": null,
                "id": 2434,
                "firstLogin": "2020-06-01 18:02:46 +0000"
            },
            {
                "clientId": "hrhtj4p8dz9wqhwtpuvg2k8ndet748vs",
                "lastLogin": "2020-06-30 18:35:38 +0000",
                "name": null,
                "id": 2817,
                "firstLogin": "2020-06-30 18:35:38 +0000"
            },
            {
                "clientId": "34way7esasgyjsq99wu7emu6wtt82j8w",
                "lastLogin": "2020-08-24 18:04:29 +0000",
                "name": null,
                "id": 4993,
                "firstLogin": "2020-08-24 18:04:29 +0000"
            }
        ],
        "emailVerified": "2020-05-29 13:49:15 +0000",
        "displayName": "August Joseph Springer",
        "id": 2433,
        "mobileNumberVerified": null,
        "externalId": null,
        "legalAcceptances": [
            {
                "clientId": "u74hp2xa4u75dq9s6wv8yyb28wkkux7m",
                "dateAccepted": "2020-04-15 21:57:46 +0000",
                "legalAcceptanceId": "privacyPolicy-v1",
                "id": 2435
            },
            {
                "clientId": "u74hp2xa4u75dq9s6wv8yyb28wkkux7m",
                "dateAccepted": "2020-04-15 21:57:46 +0000",
                "legalAcceptanceId": "termsOfService-v1",
                "id": 2436
            }
        ]
    },
    "stat": "ok"
}


Best of all, you’ll get back this information regardless of any replication lags that might be occurring. That’s because the data is being returned from the primary datastore and not the replica datastore.