More info

JUMP TO:

The Export File

A single file will be exported for each entityType configured in your expression file. The naming convention of the export file is entityType_dateTimeStamp.ext (e.g. user_20200918T071048.csv or user_20200918T071048.json).

You can optionally add a custom prefix to each file name; see the exportFilePrefix key under Configure expression file.

exportFormat: CSV

The exported csv file follows this format:

  • Each column corresponds to a data attribute defined in the expression file (entitiesToExport: attributes). The column header is the attribute name.
  • Each row below the header row contains a single user record.

Example:

uuid email givenName familyName birthday
23291df7-5796-4c6f-8afd-a684b2ba7576 John.Moore_908@test.com John Moore 2/14/1991
157e6ef1-1513-403e-a86c-47849dab7478 Tiffany.Henson_608@test.com Tiffany Henson 4/18/1981
fbc382fb-3703-4bb6-8244-0d3794d41fd3 Sonia.Hopkins_492@test.com Sonia Hopkins 11/26/1956

exportFormat: JSON

The exported json file contains an array with an object for each user record.

Example:

[
  {
    "givenName" : "John",
    "email" : "John.Moore_908@test.com",
    "uuid" : "23291df7-5796-4c6f-8afd-a684b2ba7576",
    "familyName" : "Moore",
    "birthday" : "1991-02-14"
  },
  {
    "givenName" : "Tiffany",
    "email" : "Tiffany.Henson_608@test.com",
    "uuid" : "157e6ef1-1513-403e-a86c-47849dab7478",
    "familyName" : "Henson",
    "birthday" : "1981-04-18"
  },
  {
    "givenName" : "Sonia",
    "email" : "Sonia.Hopkins_492@test.com",
    "uuid" : "fbc382fb-3703-4bb6-8244-0d3794d41fd3",
    "familyName" : "Hopkins",
    "birthday" : "1956-11-26"
  }
]

Object and plural attributes

You can configure object and plural attributes in your expression file a couple of different ways, depending on the information you want in the export file.

Objects

Configure the name of the parent object to receive all attributes within the object. In a csv file, each attribute within the object will be placed in its own column and the column headers will identify each attribute in dot notation.

  • Example configuration: \"primaryAddress\"
  • Example result in CSV export file:
    primaryAddress.address primaryAddress.city primaryAddress.state primaryAddress.zip primaryAddress.country
    23 Street Ave | ityville | H | 2345 | SA |
  • Example result in JSON export file:
"primaryAddress" : {
  "address" : "123 Street Ave",
  "city" : "Cityville",
  "state" : "OH",
  "zip" : "12345",
  "country" : "USA"
},

Configure a child attribute in dot notation to receive only that attribute, rather than every attribute contained in the object.

  • Example configuration: \"primaryAddress.zip\"
  • Example result in CSV export file:
    primaryAddress.zip
    2345 |
  • Example result in JSON export file:
"primaryAddress" : {
  "zip" : "12345"
},

Plurals

Configure the name of the parent plural to receive the entire contents of the plural. In a csv file, plural data is exported as JSON within a single column.

  • Example configuration: \"legalAcceptances\"
  • Example result in CSV export file:
    legalAcceptances
    {“clientId”:“a7nnd3wp5qaemt2wu4aqxe6xrh9jq3jd”,“dateAccepted”:“2018-01-14 03:12:04 +0000”,“legalAcceptanceId”:“privacy_policy_v1”,“id”:4738},{“clientId”:“a7nnd3wp5qaemt2wu4aqxe6xrh9jq3jd”,“dateAccepted”:“2018-01-14 03:12:04 +0000”,“legalAcceptanceId”:“terms_v1”,“id”:4739}] |
  • Example result in JSON export file:
"legalAcceptances" : [
  {
    "clientId" : "a7nnd3wp5qaemt2wu4aqxe6xrh9jq3jd",
    "dateAccepted" : "2018-01-14 03:12:04 +0000",
    "legalAcceptanceId" : "privacy_policy_v1",
    "id" : 4738
  },
  {
    "clientId" : "a7nnd3wp5qaemt2wu4aqxe6xrh9jq3jd",
    "dateAccepted" : "2018-01-14 03:12:04 +0000",
    "legalAcceptanceId" : "terms_v1",
    "id" : 4739
  }
],

Configure a child attribute in dot notation to receive only that attribute within the plural JSON, rather than every attribute contained in the plural. Note that the plural entry id is always returned along with the specified child attribute.

  • Example configuration: \"legalAcceptances.legalAcceptanceId\"
  • Example result in CSV export file:
    legalAcceptances
    {“legalAcceptanceId”:“privacy_policy_v1”,“id”:4738},{“legalAcceptanceId”:“terms_v1”,“id”:4739}] |
  • Example result in JSON export file:
"legalAcceptances" : [
  {
    "legalAcceptanceId" : "privacy_policy_v1",
    "id" : 4738
  },
  {
    "legalAcceptanceId" : "terms_v1",
    "id" : 4739
  }
],

The Last Run Date

The first time the integration runs, it will export ALL records in the specified entity type(s). At the end of the first run, a file called DataExportLastRun.json is created in your project folder (you can find it in the Manager tab). This is the file which the integration will refer to and write to each time it runs thereafter, in order to grab only the new and updated records since the previous run.

If you ever need to reset and export ALL records again, simply delete this file. The absence of the file lets the integration know that there is no LastRunDate, therefore it will act as if it is the very first run and export all user records.

If you accidentally delete this file but do NOT want to reset and export all records, you can create it with the same file name on your computer. The content of the file should look like this:

[
  {
    "EntityType" : "user",
    "ExportFilePrefix" : "",
    "LastRunDate" : "'2022-08-25T19:13:08.682Z'"
  },
  {
    "EntityType" : "user_other",
    "ExportFilePrefix" : "",
    "LastRunDate" : "'2022-08-25T19:13:13.817Z'"
  }
]

You would need to:

  • Revise the number of objects in the file to match your actual export(s)
  • Replace the values (shown above in green) with the real entity type name(s), export file prefix(es) if applicable, and the real last run date/time(s)
  • Upload the file to your project folder before the next scheduled run begins

Troubleshooting Tips

Rate limit errors

If you run into API rate limit errors, these are likely coming from the Export Data snap. Export Data snap in SnapLogic pattern The Export Data snap is the first snap in the DataExport_ProcessAndSave pipeline. Open this pipeline in the Designer tab, click on this snap to open it, and increase the value of the API interval (ms) setting. This will increase the time, in milliseconds, between calls being made to the Identity Cloud. The default is 1000 which should suffice in most cases. However if you’re running into API limit errors, try increasing this to solve the error. Export Data snap settings in SnapLogic

If you’re getting a timeout error, you may also want to the check the Timeout setting in your scheduled task configuration. Make sure this setting is blank.

Scheduled task disabled

If your pipelines are deleted at any point (perhaps you deleted your pipeline to get a fresh copy from the pattern), this will disable your related scheduled task. SnapLogic will send you an email alert if this happens. In order to get the scheduled task working again, you must do two things:

  1. Re-select the Pipeline in the scheduled task settings (select DataExport) Re-select Pipeline in scheduled task
  2. Re-enable the task:
    • Go to the Tasks tab for your project folder
    • See the Enabled column. If your task has been disabled, you’ll see a No here.
    • Check the box next to the disabled task
    • Click Enable in the upper right corner
    • This should set the Enabled column value from No to Yes Re-enable scheduled task in SnapLogic