The Ingestion Challenge at Scale
Every data-driven organization eventually reaches the same turning point: data arrives more quickly, from more sources, and in more formats, stretching the team responsible for understanding it thin. Whether it's streaming events from Kafka, flat files landing in cloud storage, or API pulls from SaaS platforms, the volume and diversity of incoming data pose a crucial question for your data ingestion strategy:
Who owns ingestion, and how should it be built?
This isn't just a technical decision. It involves team structure, data governance, velocity, and long-term maintainability. There's no single right answer, but understanding the trade-offs is important for building a data ingestion strategy that scales.
Option 1: Let Domain Teams Own Their Ingestion
In a medallion architecture, domain product owners usually own their bronze, silver, and gold layers. A natural extension of that ownership is domain-owned ingestion, letting them control how data first enters the bronze layer.
What You Gain
Speed and autonomy. Domain teams understand their source systems better than anyone else. They can act quickly without waiting for a centralized platform team.
Contextual decisions. The team closest to the data makes the best choices about schema management, error tolerance, and refresh schedules.
Clear accountability. When something breaks, there is no doubt about who owns the fix.
What You Need to Work Out
Shared standards. Without guardrails, you'll end up with five teams solving the same problem in five different ways — differences in logging, retry logic, and metadata conventions.
Tooling consistency. If one team uses Python scripts, another uses Spark, and a third uses a managed connector, your operational overhead increases.
Governance integration. Enforce data lineage, quality checks, and access controls uniformly, no matter who builds the pipeline.
What Might Slow You Down
Redundant efforts. Teams often duplicate common tasks such as CDC processing, file monitoring, or schema changes.
Skill shortages. Not all domain teams have strong data engineering expertise, which can lead to reliability and performance issues.
Oversights in governance. Decentralized data ingestion can create shadow pipelines that bypass cataloging, classification, and compliance standards.
A critical pitfall in decentralized ingestion is failing to define a schema contract for evolving data. Without a clear policy for handling dropped columns or type changes, domain teams risk breaking downstream analytics. Organizations should evaluate tools based on how they specifically manage schema evolution, such as options to freeze, discard, or automatically evolve schemas when malformed rows or new fields appear.
Option 2: Build a Centralized Data Framework
The alternative is to invest in a centralized data framework—a platform layer that standardizes how data enters the lakehouse. This could be a set of reusable components, templates, and services that domain teams configure instead of building from scratch.
What You Gain
Consistency at scale. Every pipeline follows the same patterns for error handling, monitoring, metadata capture, and schema management.
Built-in governance. The framework integrates data classification, lineage tracking, and quality validation from the start, rather than adding them later.
Operational efficiency. One team manages the platform; domain teams focus on configuration and business logic.
What You Need to Work Out
Scope management. A framework that tries to handle every ingestion pattern- streaming, batch, CDC, API, and file-based- can become unwieldy. Focus on the 80% case.
Adoption friction. If the framework is too rigid or complex, domain teams will find workarounds instead of following it.
Staffing the platform team. Building and maintaining a solid ingestion framework requires dedicated, senior engineering talent. That's a significant investment.
What Might Slow You Down
Bottlenecks. If every new source requires the platform team's involvement, it creates a queue, which slows progress.
Feature creep. Frameworks tend to expand. What starts as a simple abstraction can turn into a separate product with its own backlog, bugs, and technical debt.
Delayed time to value. Building the framework takes time. Meanwhile, the business still needs data to flow.
Recent benchmarks reveal the high cost of maintaining custom-built frameworks. A 2026 report found that organizations spend approximately $2.2 million per year in engineering labor alone on pipeline maintenance, a burden that is notably higher for teams relying on DIY or legacy integration systems rather than managed frameworks.
Option 3: Adopt an Open-Source Orchestration Platform
Products such as Apache Airflow, Dagster, Prefect, and Mage offer open-source orchestration platforms with ready-to-use features that can accelerate your ingestion plan. Instead of starting from scratch, you're building on a trusted platform.
What You Gain
Mature ecosystem. Airflow alone includes thousands of operators and integrations, so you don't have to reinvent connectors for common sources.
Community and talent pool. Engineers are already familiar with these tools, which speeds up hiring and onboarding.
Built-in observability. Most orchestration platforms come with logging, alerting, retry logic, and dependency management out of the box.
What You Need to Work Out
Customization versus convention. Open-source tools are flexible, which means your team still needs to establish conventions for their use—such as DAG structure, naming, testing, and deployment.
Governance layer. Orchestration tools handle workflow execution but do not cover data governance. You will still need to integrate with a data catalog, quality framework, and access control system.
Managed versus self-hosted. Running Airflow yourself is a significant operational challenge. Managed options like Astronomer, MWAA, and Cloud Composer reduce that burden but increase costs and impose constraints.
What Might Slow You Down
Operational complexity. Self-managed deployments require infrastructure expertise—such as Kubernetes, worker scaling, database backends, and log management.
Upgrade cycles. Open-source projects evolve rapidly. Staying current necessitates continuous testing and migration.
Lock-in by convention. Although the tool is open source, substantial investment in a platform's specific patterns can make switching costly later.
The Governance Thread
No matter which approach you choose, data governance should always be a priority. At the ingestion layer, governance involves:
Data classification. Identifying what's sensitive before it reaches bronze—such as PII, PHI, and financial data—ensures proper handling from the beginning.
Lineage capture. Tracking where data originated, how it was transformed, and where it goes is essential for compliance, troubleshooting, and building trust.
Quality contracts. Setting clear expectations for freshness, completeness, and schema adherence during ingestion, not afterward.
Access control. Making sure raw data in bronze is properly restricted, especially when it contains unmasked sensitive information.
As of June 2026, data silos remain the top challenge for 68% of organizations, a figure that has increased by 7% year-over-year. This suggests that without integrated governance and quality checks at the point of ingestion, the problem scales alongside data growth rather than resolving itself.
The best data ingestion strategy integrates governance into the pipeline rather than treating it as a gate. When governance is built into the ingestion process, teams using it don't notice it, which is exactly the point.
Finding Your Path Forward
Most organizations don't depend on a single pure approach. In practice, it often looks more like a hybrid.
A platform team offers a centralized data framework with shared tooling, standards, and governance guidelines.
Domain teams run their pipelines within these standards by configuring them rather than building from scratch.
An open-source orchestration platform—often Airflow or Dagster—acts as the execution backbone.
A governance layer is integrated directly into the platform rather than being added afterward.
The key is to be deliberate about where you set boundaries and to revisit them as your data volumes, team size, and organizational maturity grow.
Questions Worth Asking
Before committing to a direction, consider:
How many distinct source systems are you ingesting from today? How many in 12 months?
Do your domain teams have the engineering capacity to ensure ingestion reliability?
What does your data governance strategy require at the point of data entry?
Are you optimizing for speed to the first pipeline, or for consistency across dozens?
What's your tolerance for managing infrastructure versus purchasing managed services?
The answers will help you find the right balance of ownership, tools, and data governance for your organization — not someone else's.
Further reading
AWS Well-Architected DevOps Guidance — Reference definitions for key performance metrics, including ingestion rate and processing latency.
Snowflake Engineering Blog — Technical benchmarks for high-volume ingestion paths and client-side decompression costs.
Scalable Ingestion Design Principles — An overview of leveraging distributed architectures like Kafka and Spark for real-time collection.
Observability Pricing Benchmarks 2026 — A comparison of ingest-based versus workload-based pricing models for large-scale data monitoring.