Validate & execute

At this point, the Data Export integration is configured and ready to execute. Before setting up a scheduled task to run the export on a recurring basis, we recommend you validate and execute the integration manually.

Validate

Validating a pipeline allows you to uncover and address issues before running an actual export.

To validate the integration, navigate to the Designer tab in SnapLogic.

Then:

  1. In the left navigation, go to the Pipelines tab
  2. Click to open the Identity Cloud Data Export project space
  3. Click to open your project folder within the space
  4. Click the DataExport pipeline to open it in the main panel

Let’s validate this pipeline to see if there are any configuration issues we may need to resolve:

  1. Click the Validate Pipeline icon in the upper right corner. This icon appears as a “V” within a cog. Validate Pipeline in SnapLogic
  2. As the validation runs, all snaps will start out grey, and will turn light green as they successfully process.

If there are any issues:

  • A snap may turn red. Click on the red snap to view the error details.
  • A window may appear with an error message. If you scroll through the list of snaps in this window, you’ll see the one that errored has a red Show Details… link you can click for more information about the issue and what to fix.

Once you fix the issue, try validating the pipeline again using the Retry link. This ensures your cache is cleared and any configuration changes are pulled in before the next validation: Validation retry in SnapLogic

Execute

Once your pipeline validates successfully, it is ready to be executed. Even if you plan to run this integration on a scheduled basis (e.g. once a day at midnight), we recommend executing it the first time manually to observe and correct any remaining issues.

Note that the first time the pipeline is executed, ALL USER RECORDS in the configured entity type(s) will be exported to your receiver account. The more user records you have, and the more attributes being exported per record, the longer the export will take to finish.

As a reference point, exporting about 1 million records with about 20 attributes each may take about 30 minutes.

To execute the integration:

  1. Click the Execute Pipeline icon in the upper right corner. This icon appears as a play button within a circle. Execute Pipeline in SnapLogic
  2. As the execution runs, all snaps will turn yellow when they are processing and dark green when they successfully complete.
  3. When it appears the execution has completed successfully, go to your sFTP server or S3 bucket to download and inspect the resulting export file.

If there are any issues:

  • A snap may turn red. Click on the red snap to view the error details.
  • A window may appear with an error message. If you scroll through the list of snaps in this window, you’ll see the one that errored has a red Show Details… link you can click for more information about the issue and what to fix.

Once you fix the issue, try executing the pipeline again. You may need to refresh the page before running the next execution, as previous configurations from the Manager tab can be cached in the Designer tab.

NOTE! The first time your pipeline is executed, it will create a file called DataExportLastRun.json in your project folder. (You can find it in the Manager tab.) This is an important file that ensures future executions only pull new or updated records since the previous run. See The Last Run Date for more information.