More info
| JUMP TO: |
|---|
The Export File
A single CSV file will be exported for each entityType configured in your expression file. The format of the export file name is entityType_dateTimeStamp.csv (e.g. user_20200918T071048.csv).
You can optionally add a custom prefix to each file name; see the exportFilePrefix key under Configure expression file.
The exported CSV file follows this format:
- Each column corresponds to a data attribute defined in the expression file (
entitiesToExport:attributes). The column header is the attribute name. - Each row below the header row contains a single user record.
Example:
| uuid | givenName | familyName | birthday | |
|---|---|---|---|---|
| 23291df7-5796-4c6f-8afd-a684b2ba7576 | John.Moore_908@test.com | John | Moore | 2/14/1991 |
| 157e6ef1-1513-403e-a86c-47849dab7478 | Tiffany.Henson_608@test.com | Tiffany | Henson | 4/18/1981 |
| fbc382fb-3703-4bb6-8244-0d3794d41fd3 | Sonia.Hopkins_492@test.com | Sonia | Hopkins | 11/26/1956 |
Object and plural attributes
You can configure object and plural attributes in your expression file a couple of different ways, depending on the information you want in the export file.
Objects
Configure the name of the parent object to receive all attributes within the object. Each attribute within the object will be placed in its own column in the CSV and the column headers will identify each attribute in dot notation.
- Example configuration:
\"primaryAddress\" - Example result in export file:
primaryAddress.address primaryAddress.city primaryAddress.state primaryAddress.zip primaryAddress.country 123 Street Ave Cityville OH 12345 USA
Configure a child attribute in dot notation to receive only that attribute, rather than every attribute contained in the object.
- Example configuration:
\"primaryAddress.zip\" - Example result in export file:
primaryAddress.zip 12345
Plurals
Configure the name of the parent plural to receive the entire contents of the plural. Plural data is exported as JSON within a single column.
- Example configuration:
\"legalAcceptances\" - Example result in export file:
legalAcceptances [{“clientId”:“a7nnd3wp5qaemt2wu4aqxe6xrh9jq3jd”,“dateAccepted”:“2018-01-14 03:12:04 +0000”,“legalAcceptanceId”:“privacy_policy_v1”,“id”:4738},{“clientId”:“a7nnd3wp5qaemt2wu4aqxe6xrh9jq3jd”,“dateAccepted”:“2018-01-14 03:12:04 +0000”,“legalAcceptanceId”:“terms_v1”,“id”:4739}]
Configure a child attribute in dot notation to receive only that attribute within the plural JSON, rather than every attribute contained in the plural. Note that the plural entry id is always returned along with the specified child attribute.
- Example configuration:
\"legalAcceptances.legalAcceptanceId\" - Example result in export file:
legalAcceptances [{“legalAcceptanceId”:“privacy_policy_v1”,“id”:4738},{“legalAcceptanceId”:“terms_v1”,“id”:4739}]
The Last Run Date
The first time the integration runs, it will export ALL records in the specified entity type(s). At the end of the first run, a file called DataExportLastRun.json is created in your project folder (you can find it in the Manager tab). This is the file which the integration will refer to and write to each time it runs thereafter, in order to grab only the new and updated records since the previous run.
If you ever need to reset and export ALL records again, simply delete this file. The absence of the file lets the integration know that there is no LastRunDate, therefore it will act as if it is the very first run and export all user records.
If you accidentally delete this file but do NOT want to reset and export all records, you can create it with the same file name on your computer. The content of the file should look like this:
[{"EntityType":"user","LastRunDate":"'2020-10-07T17:57:22.713Z'"},{"EntityType":"user_other","LastRunDate":"'2020-10-07T17:57:24.045Z'"}]You’ll need to replace the EntityType and LastRunDate values (above in orange) with the real entity type name(s) and the real last run date/time(s), then upload the file to your project folder before the next scheduled run begins.
Troubleshooting Tips
Rate limit errors
If you run into API rate limit errors, these are likely coming from the Export Data snap.
The Export Data snap is the first snap in the DataExport_ProcessAndSave pipeline. Open this pipeline in the Designer tab, click on this snap to open it, and increase the value of the API interval (ms) setting. This will increase the time, in milliseconds, between calls being made to the Identity Cloud. The default is 1000 which should suffice in most cases. However if you’re running into API limit errors, try increasing this to solve the error.
If you’re getting a timeout error, you may also want to the check the Timeout setting in your scheduled task configuration. Make sure this setting is blank.
Scheduled task disabled
If your pipelines are deleted at any point (perhaps you deleted your pipeline to get a fresh copy from the pattern), this will disable your related scheduled task. SnapLogic will send you an email alert if this happens. In order to get the scheduled task working again, you must do two things:
- Re-select the Pipeline in the scheduled task settings (select DataExport)
- Re-enable the task:
- Go to the Tasks tab for your project folder
- See the Enabled column. If your task has been disabled, you’ll see a No here.
- Check the box next to the disabled task
- Click Enable in the upper right corner
- This should set the Enabled column value from No to Yes
