Setting up Code Repository for Azure Data Factory

By | July 15, 2018

Setting up Code Repository for Azure Data Factory (ADF) Pipeline - Featured

Azure Data Factory is a cloud-based data orchestration service that enables data movement and transformation. You can set up code repository for Azure Data Factory (ADF) and have an end to end integrated development and release experience. Attach to a code repository for data factory and have your configuration JSON for the dataset, linked services, and pipelines. In this post, we will quickly look how to set up the code repository for Azure Data Factory.

Code Repository for Azure Data Factory

From the Azure Portal, open the respective Azure Data Factory resource. Select the “Author & Monitor” tile under quick links for open the ADF Pipeline. Consider you have an existing ADF Pipeline. Select the “Set up Code Repository” option to open the repository settings wizard.

Code Repository for Azure Data Factory - Set up code repository

Repository settings wizard will have the option to provide the information related with data factory and code repository. At this point of type, we have the only support of Visual Studio Team Services Git as a repository.

Code Repository for Azure Data Factory - Repository Settings

Select the respective projects and details of the branch, save the information. Now you can create a new local branch, however, for now, let’s go ahead by selecting “master” branch.

Code Repository for Azure Data Factory - Select Branch

That’s it. Your repository configuration is done. You can go back to your Visual Studio Team Services, to check the pushed code.

Now, VSTS code repository now has all the JSON files for the dataset, linked services, and pipeline. You can now start doing your changes, commit it to push it back to your Azure Data Factory.

Code Repository for Azure Data Factory - Code in VSTS

From the Visual Studio Team services, you can check out all the changes history including the changes from the pipeline.

Code Repository for Azure Data Factory - History

From here onward, you can have your code with your Favorite IDE like Visual Studio or Visual Studio code to do necessary changes for your Azure Data Factory.

Local Branch for ADF and Create Pull Request

You can create pull requests and have your local branch for Azure Data Factory.  Maintain your changes in the local branches and push the code to master using a pull request.

This will enable end to end code flow management for proper release management.

Author: Abhijit Jana

Abhijit runs the Daily .NET Tips. He started this site with a vision to have a single knowledge base of .NET tips and tricks and share post that can quickly help any developers . He is a Former Microsoft ASP.NET MVP, CodeProject MVP, Mentor, Speaker, Author, Technology Evangelist and presently working as a .NET Consultant. He blogs at http://abhijitjana.net , you can follow him @AbhijitJana . He is the author of book Kinect for Windows SDK Programming Guide.

Leave a Reply