.webp?width=1921&height=622&name=usecase-banner%20(1).webp)
Key takeaway
- When a data pipeline fails across NiFi, HDFS, Spark and Airflow, who owns the fix? A co-managed L2/L3 responsibility model with a RACI and the metrics to run it.
Who is responsible when a data pipeline fails across several teams? In most enterprises, nobody is, until the incident ticket lands on someone's desk. Infrastructure owns the virtual machines, a platform team owns Kubernetes, data engineering owns the pipelines, security owns access policy and a database team owns the downstream store. A co-managed support model fixes this by assigning cross-component diagnosis to one specialist function, while the enterprise keeps ownership of its architecture, business logic and change approvals.
The 60-second brief
- The problem: business sees one failure, "the data was late". The organisation sees five teams and five queues.
- The principle: the answer to "who owns this?" should not depend on which team receives the ticket first.
- The model: the enterprise runs L1 monitoring and owns decisions. A specialist partner provides L2/L3 diagnosis across components and drives prevention.
- The tool: a written responsibility matrix covering incidents, changes, security and improvement.
- The measure: not tickets closed, but recurring failures eliminated and data delivered by its business cut-off.
Part of a series. Start with the overview: Open-Source Data Platforms on Private Cloud: Why Managed Services and Governance Must Evolve Together.
One Business Outcome, Many Technical Owners
Consider a typical open-source analytics platform on private cloud. Files arrive by secure transfer. Apache NiFi ingests them. HDFS stores raw and transformed data. Airflow schedules processing. Spark transforms. YARN allocates compute. Druid serves analytics.
Each component can be healthy on its own dashboard while the data journey fails:
- A late source file delays everything downstream.
- A congested NiFi queue slows ingestion.
- HDFS capacity pressure makes Spark jobs fail.
- YARN contention pushes a critical job past its window.
- An Airflow dependency failure skips a scheduled report.
From the business side these are one event: trusted data was not available when needed.
Green infrastructure dashboards and a missing report can both be true at the same time.
Why Full Outsourcing Is Often the Wrong Answer
Internal teams understand the business processes, the transformations and what matters at month-end. Replacing that with an external operations team adds friction and risk. Yet few internal teams have deep, current expertise in every one of Hadoop, Spark, NiFi, Airflow and Druid. Co-managed support keeps what the enterprise does best and adds what is hard to retain in-house.
The Responsibility Matrix
R = responsible, A = accountable, C = consulted, I = informed. Adapt the columns to your organisation.
| Activity | Enterprise data platform team | Enterprise infrastructure and security | Co-managed L2/L3 partner |
|---|---|---|---|
| L1 monitoring and incident detection | R, A | C | I |
| Cross-component diagnosis (L2/L3) | C | C | R, A |
| Service restoration within approved runbooks | A | C | R |
| Production change approval | R, A | C | C |
| Business logic and pipeline code | R, A | I | C |
| Operating system, virtual machines, Kubernetes | I | R, A | C |
| Access policy and security controls | C | R, A | C |
| Root-cause analysis and known-error records | A | C | R |
| Performance tuning recommendations | A | I | R |
| Preventive actions and recurring-failure removal | A | C | R |
| Lifecycle and upgrade planning | A | C | R |
| Decision to reprocess a failed batch | R, A | I | C |
Clear boundaries are not an obstacle to collaboration. They are what make collaboration dependable.
Three Things to Make Explicit
- Response versus restoration. A commitment to acknowledge an incident is not a commitment to restore service. Restoration often depends on teams the partner does not control. State each separately.
- Coverage hours. Define business-hours specialist support and, where needed, a separately arranged on-call arrangement for critical incidents.
- Emergency authority. Who may authorise a production restart, grant privileged console access during an incident, or approve an emergency change? Decide before the incident.
From Reactive Support to Platform Assurance
A platform that has the same failure every month does not have a sustainable operating model, even if every incident is resolved inside its target. Co-managed support should carry three responsibilities:
- Protect critical data journeys. Diagnose across components, with the business impact in view.
- Improve performance and resilience. Address capacity trends, scheduling, back-pressure, query latency and configuration risk before they cause incidents.
- Build operational knowledge. Turn every significant incident into a runbook, a known-error record and a preventive action. This reduces dependence on individuals.
Metrics That Matter
| Operations | Governance |
|---|---|
| Critical pipeline success rate | Privileged access reviews completed |
| Data freshness against business cut-offs | Open critical control gaps |
| Acknowledgement and restoration times | Certificate and credential hygiene |
| Recurring failure rate | Security exceptions and their age |
| Capacity headroom | Audit evidence readiness |
| Preventive actions completed | Components past community support |
Baseline these before agreeing targets. When a pipeline is late, leadership should be able to see which business process is affected, whether an approved recovery path exists, who owns the next decision and what will prevent a repeat.
Where This Model Is Tested Hardest
- Major version changes, such as NiFi 1 to NiFi 2 and Airflow 2 to Airflow 3.
- Leaving a commercial distribution. See After Cloudera: Who Operates Your Open-Source Hadoop Platform.
- Security hardening of a live platform. See the DPDP 2027 platform controls checklist.
- Introducing AI assistance into operations. See AI SRE for Data Platforms.
The Bottom Line
The aim of co-managed support is not to make an enterprise more dependent on an outside provider. It is to make the platform more resilient, more governable and less dependent on undocumented individual knowledge. Write the responsibilities down before the next incident writes them for you.
Set Up Co-Managed Support with XenonStack
XenonStack provides co-managed L2/L3 support for open-source data platforms on private cloud, covering Hadoop, Spark, NiFi, Airflow and Druid, with reliability engineering and governance enablement. Explore Data Managed Services and the Open-Source Data Platform solution.
Frequently Asked Questions
What is co-managed support for a data platform?
An operating model in which the enterprise keeps ownership, first-line monitoring and change approval, while a specialist partner provides advanced diagnosis, performance engineering and planned improvement.
What is the difference between L1, L2 and L3 support?
L1 detects and triages incidents. L2 performs deeper diagnosis and restoration using runbooks. L3 handles complex, cross-component problems, root-cause analysis and engineering-level fixes.
Who should approve production changes in a co-managed model?
The enterprise. The partner recommends and prepares changes; approval stays with the organisation that owns the platform and its risk.
How do we measure whether the model works?
By business-facing outcomes: critical pipeline success, data freshness against cut-offs, falling recurring failures and completed preventive actions.
Does co-managed support cover security hardening?
Ongoing support covers operating the approved controls. Major hardening, such as introducing new authentication or encryption, is best run as a separately scoped engineering programme.
Explore the XenonStack Platform and Agent Network
Platforms: ElixirData · ElixirClaw · ElixirHub · NexaStack · MetaSecure · Neural AI · AutonomousOps · CohortIQ · AI Aviator · SAIF Aviator
AI agents: AgentAnalyst · AgentEvaluation · AgentForce · AgentGRC · AgentHR · AgentInstruct · AgentLabel · AgentPixel · AgentQA · AgentRAI · AgentScrum · AgentSearch · AgentSketch · AgentSRE
XenonStack: xenonstack.com · xenonstack.ai · xenonstack.ae · xenonstack.jobs
Not sure which decision to hand to governed agents first? Book a Context Workshop to map one AI decision in half a day and leave with a go/no-go.
Explore the XenonStack platform family: ElixirData · ElixirClaw · ElixirHub · NexaStack · MetaSecure · Neural AI · AgentGRC · AgentSRE · AgentAnalyst · AgentInstruct · AgentRAI · AgentScrum · AgentSearch · AgentForce · AI Aviator · SAIF Aviator · CohortIQ · XenonStack AI · XenonStack UAE
