Good news! Moving your pipeline(s) is easier than you think. It can be accomplished in DevOps, but if you don’t have DevOps configured in both Data Factory and Synapse, it’s overwhelming to deal with source control. By completing the following steps, you can save time and seamlessly “move” your pipeline from Data Factory to Synapse.
Note that prior to completing these steps, you should manually create all of the linked services in Synapse that are in your Data Factory. Once those linked services are created, you are clear to get started.
Open your Azure Data Factory (ADF) and navigate to the desired pipeline.
Click on the ‘Actions' button (ellipsis) next to your pipeline.
3. From the dropdown menu, select 'Download support files'.
4. The files will be downloaded as a zip folder. Extract all files from the zip folder.
5. Locate the pipeline folder.
6. In the pipeline folder, you'll find a JSON file. Copy the contents of that JSON file to your clipboard.
7. Open up your Azure Synapse Workspace. Navigate to the Integrate tab in the left navigation menu and then select ‘Add new resource’.
8. Click the ‘Actions’ button once more, then rename the new pipeline in Azure Synapse Workspace so that it has the same exact name as the pipeline in Azure Data Factory.
In this case, we would rename ‘Pipeline1’ to ‘ExampleAPI’.
9. Once the new pipeline is created, look for the curly brackets { } button at the top right of the pipeline designer interface.
10. Click on the curly brackets button to open the JSON editor.
11. Select all the contents in the JSON editor using the Ctrl + A shortcut, and delete the existing code.
12. Paste the JSON code that you copied from the Azure Data Factory pipeline JSON file.
13. Click 'OK' to save the changes and update the pipeline with the replicated code.
14. Congratulations! You have successfully taken a pipeline that was once in Azure Data Factory and replicated the pipeline in Azure Synapse Analytics.
Ready to Migrate to Azure Synapse?
Azure Synapse Analytics offers detailed Apache Spark pools and Notebooks, which are perfect for complex operations such as data processing and advanced analytics. It’s more than worth the effort to make the move.
If you want to learn more about how Azure is reinventing data analytics, check out this blog.
If you are looking for data migration support, reach out to a data engineer today.