This blog post is part of a “Working with OAuth 2.0 APIs in Azure Data Factory” series, and you can find a list of the other posts here.
Azure Data Factory (ADF) provides built-in linked services for some vendor APIs, but I personally don’t like to use them. Why? Well, because they work great when they do but are otherwise very difficult to troubleshoot. I’ve found them to be clunky at the best of times, and they don’t always stay in sync with the vendor’s requirements. The good news is that you can roll your own, which is what I prefer to do and what I will show here.
To create a generic linked service for a REST API simply choose the “REST” type, give it a name and add your API’s base url. It is important to use “Anonymous” as authentication type, because we will send the necessary tokens in the pipeline and do not want the linked service to try and do the authentication for us. In the example below, I am going to connect to the Xero OAuth 2.0 API:
The dataset is also pretty straight forward and I prefer to use a parameter for the relative URL, so that I can reuse the same dataset and just set the URL of the API I want to call during runtime:
With the foundational items in place, we are now ready to start building our pipelines to extract data. In the next blog post we’ll look at a pipeline we can use to refresh the tokens, an important part of automating the entire process.