Skip to content

Azure Data Factory

Azure Data Factory (ADF) is a cloud-based data integration service. It allows you to create, schedule, and manage data pipelines that move and transform data from various sources to different destinations (called sinks).

flowchart LR
    Source-->|Data|ADF;
    ADF-->|Data|Sink;

At Power Partners, we use the Data Factory mainly to fetch data (e.g. from a database or via an API) and store it into a DWH. Also, our financial accelerator product uses Azure Data Factory to trigger Azure Functions.

Here are some key features and concepts related to Azure Data Factory:

  • Pipelines: A pipeline consists of activities that perform specific tasks, such as data ingestion, data transformation, and data movement.
  • Activities: Activities are the building blocks of pipelines. They represent individual tasks within a pipeline, such as copying data from a source to a destination, running a dataflow, etc.
  • Data Sources and Sinks: connect to APIs, Azure Blob Storage, Azure SQL Database, Azure Data Lake Storage, on-premises databases, and more.
  • Data Flow: Use Data Flows to transform data (filter, join, etc.), a bit similar to Power Query and/or SQL scripts. Data flows can be developed with a visual programming language, or also with Power Query. Though Power Query in Data Flows does not support all operations (such as e.g. pivots).
  • Data Movement: Move data between different systems
  • Monitoring and Management
  • Integration with Azure Services: Data Factory integrates with other Azure services, such as Azure Functions or Azure Machine Learning.

Costs

Cost are calculated by usage. See here for a calculator: https://azure.microsoft.com/en-us/pricing/calculator/

Typically, the costs for our clients are below 20 EUR per month.

DIU

Many services are calculated with so-called DIUs (Data Integration Units). This is a compbination of CPU, memory, and network. See here for details: https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance#data-integration-units

Difference to Synapse, Fabric, SSIS, Databricks, Spark, Air Flow, etc.

There is a big overlap, and Microsoft's data strategy can be confusing. Here are some key hints:

  • Synapse can be seen as an ELT platform, while Azure Data Factory can be seen as an ETL platform. Though, there are big overlaps.
  • SSIS can be seen as the on-prem predecessor to Azure Data Factory
  • Fabric includes Azure Data Factory and integrates it (and many other services) in the Power BI platform.
  • Databricks is both a stand-alone data platform, but it is also integrated as a first class citizen in Azure Data Factory.
  • Air Flow is an open source (apache) data orchestration platform. There are big overlaps with Azure Data Factory. However, Azure Data Factory now includes Air Flow as a first class citizen, so that Air Flow workloads can be deployed as Activities. Currently, (June 2023) we are not using this at Power Partners, though it might be an interesting option to deploy our python API connectors (instead of Azure Functions).

Accessing On-Premise Ressources

We can install a self-hosted Integration Runtime on prem. This will give us access to on-prem ressources such as:

  • locally installed APIs (though not through AF, see below for details)
  • databases (for some db makes, you also need to install ODBC drivers)

It is important that the IR is installed on a server that has access to the target. The connection to the ADF is done automatically under the hood.

Azure Function

In case you call an Azure Function from the Azure Data Factory, you can set a self-hosted IR when setting up the AF service in ADF. However, this will not propagate the network of the IR to the AF.

To access local ressources with an Azure function, you need to install the Hybrid Connection Manager, which relays the calls to the on-prem network. For more details, see Azure Function.