Key takeaway
- Airflow vs Azure Data Factory in 2027: Airflow 3, Fabric Data Factory and managed Airflow, a decision table, governance, cost and when to use both.
Azure Data Factory (ADF) and Apache Airflow solve different halves of the same problem. ADF is Microsoft's managed, low-code service for moving and transforming data at scale, with built-in connectors, mapping data flows and integration runtimes. Airflow is an open-source orchestrator where pipelines are Python code (DAGs) that can call any system, including ADF. Choose ADF when most data lives in Azure and teams want visual pipelines; choose Airflow when you need code-first, cross-cloud orchestration; many enterprises use both.
Two things changed the comparison in 2025 and 2026. Microsoft now calls Data Factory in Microsoft Fabric the next generation of ADF, and offers managed Airflow there as the Apache Airflow job. Airflow itself moved to version 3: 3.0 shipped on 22 April 2025, Airflow 2 reached end of life on 22 April 2026, and the current release is 3.4.0. For Airflow basics, see our Apache Airflow guide; this post focuses on the decision between the two.
What is the difference between Azure Data Factory and Apache Airflow?
ADF is a data integration service. According to Microsoft's What is Azure Data Factory? page, it is a managed cloud service for hybrid ETL, ELT and data integration, built from pipelines, activities, datasets, linked services, mapping data flows, integration runtimes and triggers. Mapping data flows run on Spark clusters that ADF starts and stops for you.
Airflow is a workflow orchestrator. It schedules and monitors tasks arranged as directed acyclic graphs, written in Python, and hands the heavy data work to other systems through operators and hooks. Its strength is control flow and dependencies across many tools, rather than moving data itself. Our guide to workflow orchestration covers the wider category.
| Dimension | Azure Data Factory | Apache Airflow |
| Primary job | Data movement and transformation (ETL and ELT) | Orchestration and scheduling of tasks across systems |
| How pipelines are built | Visual designer, JSON definitions, mapping data flows | Python DAG files under version control |
| Hosting | Fully managed Azure service | Self-hosted, or managed by a cloud or vendor (including the Fabric Apache Airflow job) |
| Connectivity | Built-in connectors for Azure, databases, files, SaaS and generic protocols | Provider packages with operators and hooks; custom operators for anything else |
| Compute for transforms | Spark-based data flows, or external activities on Databricks, HDInsight and others | Runs tasks on workers; heavy compute is usually delegated to Spark, warehouses or other services |
| Security model | Microsoft Entra ID and Azure role-based access control | Depends on deployment; you configure authentication, secrets backends and network controls |
| Monitoring | Azure Monitor, health panels, API and PowerShell | Airflow UI (rewritten in React in 3.0), logs and metrics you export |
| Licence and cost | Pay as you go for orchestration, data movement and data flow runs | Open source; you pay for infrastructure or a managed service |
Sources: Microsoft Learn and airflow.apache.org, read 11 November 2026.
What has changed in 2025 and 2026?
Airflow 3 is now the only supported major version
The Airflow supported versions page lists Airflow 3 as the maintained line, first released on 22 April 2025, with 3.4.0 as the current release. Airflow 2 entered limited maintenance on 22 October 2025 and reached end of life on 22 April 2026. The Airflow 3.0 release post lists the main changes: DAG versioning (a run completes on the version it started with), a Task Execution Interface that moves Airflow towards a client-server design, an Edge Executor for running tasks on edge devices, Datasets evolving into Assets, event-driven scheduling, backfills run inside the scheduler, and a new UI built on React and FastAPI.
Fabric Data Factory is Microsoft's next generation of ADF
Microsoft Learn now opens ADF pages with a note that Data Factory in Microsoft Fabric is the next generation of Azure Data Factory, with a simpler architecture, built-in AI and new features, and invites existing ADF workloads to upgrade. Fabric Data Factory offers pipelines, dbt jobs, Copilot for building pipelines in natural language, and Apache Airflow jobs for code-first orchestration. ADF itself remains a supported Azure service.
Managed Airflow moved from ADF to Fabric
Microsoft describes the Apache Airflow job in Fabric as the next generation of ADF's Workflow Orchestration Manager, and publishes a migration guide for teams moving DAGs across. Compared with Workflow Orchestration Manager, the Fabric job adds high availability, deferrable operators, autoscale and pause and resume, but Microsoft lists diagnostic logs and metrics and Blob Storage as not supported, and private networks and virtual networks are not yet supported. Billing is by pool uptime; a Starter pool shuts down after 20 minutes of inactivity (Apache Airflow job pricing).
When should you choose Azure Data Factory, Airflow or both?
| Situation | Best fit | Why | Status to check |
| Most sources and targets are in Azure; analysts and data engineers prefer low code | Azure Data Factory, or Fabric Data Factory for new builds | Managed connectors, data flows and Azure security with little code to maintain | ADF supported; Microsoft steers new work to Fabric |
| Already on Microsoft Fabric for lakehouse, warehouse and Power BI | Fabric Data Factory pipelines, plus Apache Airflow jobs where code is needed | One platform, one capacity model, Copilot in pipelines | Airflow jobs lack private network support today |
| Pipelines span AWS, Google Cloud, SaaS and on-premises tools | Apache Airflow (self-hosted or a managed service) | Python DAGs and provider packages work across clouds | Run Airflow 3; Airflow 2 is end of life |
| Complex dependencies, branching, backfills and ML workflows | Apache Airflow | Code-level control of dependencies, retries and backfills, now run in the scheduler | Airflow 3.x |
| Heavy Azure data movement inside a wider cross-platform workflow | Both: Airflow as the control plane, ADF pipelines as tasks | Airflow triggers and tracks ADF runs with the Azure provider | Provider supports deferrable operators |
| Strict network isolation for managed Airflow | Self-hosted Airflow on AKS or VMs, or another managed Airflow service | Fabric Airflow jobs do not support private or virtual networks yet | Recheck Fabric release notes |
Our view: for a Microsoft-centred estate, default to Fabric Data Factory for new pipelines and keep ADF running where it works; bring in Airflow when orchestration crosses clouds or needs code-level control.
How do Azure Data Factory and Airflow work together?
The Microsoft Azure provider for Airflow includes AzureDataFactoryRunPipelineOperator to start an ADF pipeline and wait for it to finish, and AzureDataFactoryPipelineRunStatusSensor to check a run's status. Both can run in deferrable mode, which frees worker slots while ADF does the work. This gives a clean split:
- ADF or Fabric pipelines do the data work: copy activities, data flows, SSIS packages on a managed integration runtime.
- Airflow owns the end-to-end schedule: it waits for upstream events, triggers ADF, then runs dbt, Spark, ML training or a reverse ETL step outside Azure.
- Status flows back to one place: the Airflow UI shows the whole chain, while Azure Monitor keeps ADF-level detail.
When those pipelines feed AI agents rather than dashboards, the orchestration layer needs governance too. ElixirClaw is an agentic OS platform to build, orchestrate and govern AI agents and automation workflows while keeping humans in control.
How do governance, security and cost compare?
| Area | Azure Data Factory | Apache Airflow | What to decide |
| Identity and access | Microsoft Entra ID and Azure RBAC built in | Configured per deployment; managed services add their own identity integration | Who can edit, trigger and approve pipelines |
| Secrets | Linked services hold connection details; keep credentials in a managed secrets store | Secrets backends; Fabric Airflow jobs support Azure Key Vault as a backend | One secrets store across both tools |
| Change control | Git integration and CI/CD with Azure DevOps or GitHub | DAGs live in Git; DAG versioning in Airflow 3 ties each run to its code version | Review and release process for pipeline code |
| Lifecycle risk | Watch for connector deprecations and the move to Fabric | Airflow 2 is end of life since 22 April 2026; upgrade to 3.x | An owner for upgrades on each platform |
| Cost drivers | Pipeline orchestration and execution, data flow run time and debugging, Data Factory operations | Infrastructure or managed service fees; Fabric Airflow jobs bill by pool uptime in capacity units | Model cost per pipeline run before migrating |
| Operations | Azure Monitor alerts and health panels | Scheduler health, worker capacity, metadata database, log storage | Who is on call when a run fails at 2 a.m. |
Failed runs, retries and late data are where most pipeline cost and risk sit. AgentSRE uses a fleet of AI agents to automate detection, root cause analysis and remediation for incident management and observability. For data quality checks inside ADF itself, see our guide to automating data quality in Azure Data Factory.
How should you adopt or migrate?
- Inventory pipelines. List every ADF pipeline, Workflow Orchestration Manager environment and Airflow deployment, with its version, owner, schedule and downstream users.
- Retire Airflow 2. Plan the move to Airflow 3.x first; Airflow 2 no longer receives fixes. Test DAGs against the Task SDK and the new UI.
- Decide the Microsoft target. Keep stable ADF pipelines, build new ones in Fabric Data Factory if you are on Fabric, and move Workflow Orchestration Manager DAGs to the Fabric Apache Airflow job using Microsoft's migration guide.
- Set the control plane. If workflows cross clouds, make Airflow the single scheduler and call ADF or Fabric pipelines from it.
- Add governance. Central secrets, Git-based change control, run-level logging and alerting, and a named owner per pipeline.
- Measure. Track run success rate, time to recover, cost per run and data freshness before and after the change.
What are the trade-offs, and when should you not use each?
- Do not use ADF as your only orchestrator when most steps run outside Azure or need custom Python logic; the visual model becomes hard to maintain.
- Do not self-host Airflow if you have no team to run the scheduler, workers and metadata database; a managed service or ADF is likely to cost less overall.
- Do not start new Airflow 2 projects. It reached end of life on 22 April 2026.
- Do not pick the Fabric Apache Airflow job where private network isolation is mandatory, until Microsoft adds that support.
- Do not run both without a clear split. Two schedulers triggering the same jobs is a common cause of duplicate runs and missed dependencies.
The Azure Architecture Center guide to choosing a pipeline orchestration technology is a useful neutral checklist alongside this table.
To map one pipeline decision your agents should make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop.
Related on XenonStack
- DataOps AI agents: pipelines that monitor and fix themselves.
- ElixirData: governed context for enterprise AI agents.
- Agentic AI for data management: warehousing and pipelines.
What should you plan for in 2027?
- 31 March 2027: Microsoft lists Azure Data Factory Memory Optimized Data Flows as retiring on this date. Check any data flows in your estate that still use that compute type.
- Airflow 3 dates: the Airflow supported versions page lists Airflow 3 as maintained, with its limited maintenance and end of life dates still to be decided. Airflow 2 is already end of life, so plan on 3.x.
- Name an owner: give each platform one person responsible for tracking these notices, whether the pipelines run in ADF, Fabric or Airflow.
Frequently asked questions about Airflow vs Azure Data Factory
Is Airflow better than Azure Data Factory?
Neither is better in general. ADF is stronger for managed, low-code data movement and transformation in Azure. Airflow is stronger for code-first orchestration across clouds and tools. Many enterprises use Airflow to trigger ADF pipelines.
Can Apache Airflow run Azure Data Factory pipelines?
Yes. The Microsoft Azure provider for Airflow includes AzureDataFactoryRunPipelineOperator to run an ADF pipeline and AzureDataFactoryPipelineRunStatusSensor to check its status, and both support deferrable mode.
Is Azure Data Factory being replaced by Fabric?
Microsoft describes Data Factory in Microsoft Fabric as the next generation of Azure Data Factory and recommends new users start there, with an upgrade path for existing ADF workloads. ADF remains a supported Azure service.
What happened to managed Airflow in Azure Data Factory?
Microsoft describes the Apache Airflow job in Microsoft Fabric as the next generation of ADF's Workflow Orchestration Manager and publishes a guide to migrate DAGs to it.
Which Airflow version should enterprises run in 2027?
Airflow 3. Version 3.0 was released on 22 April 2025 and 3.4.0 is the current release. Airflow 2 reached end of life on 22 April 2026.
Next Steps in Implementing Data Pipelines with Azure and Airflow
Talk to our experts about implementing data pipeline orchestration with Azure Data Factory and Apache Airflow. Learn how enterprises streamline ETL workflows, automate data integration, and enhance operational efficiency with cloud-native and open-source solutions. Improve data pipeline reliability, scalability, and performance with AI-driven automation.
Follow XenonStack on LinkedIn: XenonStack on LinkedIn for new guides on agentic AI, data and platform engineering.
Explore the XenonStack platform family: ElixirData · ElixirClaw · ElixirHub · NexaStack · MetaSecure · Neural AI · AgentGRC · AgentSRE · AgentAnalyst · AgentInstruct · AgentRAI · AgentScrum · AgentSearch · AgentForce · AI Aviator · SAIF Aviator · CohortIQ · XenonStack AI · XenonStack UAE