The Architecture-First Trap
The conference room buzzes with energy. Your team is debating the future data platform—Snowflake's simplicity versus Databricks' ML capabilities versus building custom on AWS for ultimate flexibility. Engineers sketch architectures on whiteboards. Budget models circulate. The CTO wants a decision by Friday.
There's just one problem: no one has clearly defined what you're actually building.
This scenario plays out daily in organizations worldwide. The excitement of modern data platforms—their capabilities, promise, and vendor presentations—drives decisions before teams fully understand their actual needs. The platform becomes the strategy rather than an outcome of strategy.
I've watched this pattern create millions in technical debt. In one case, a financial services firm spent eighteen months building a custom data lake on AWS, only to realize that their core need was ACID-compliant transaction processing, which a managed platform would have delivered in weeks. In another, an organization invested heavily in Databricks for "future ML capabilities" that never materialized, while their analysts struggled with basic reporting that Snowflake would have solved immediately.
The truth is simple but often ignored: platform selection is the answer, not the starting question.
What Requirements Really Mean in Data Engineering
When data leaders talk about "requirements gathering," they often mean collecting a list of reports or dashboards stakeholders want. That's not requirements analysis—that's order-taking.
True requirements discovery in data engineering encompasses several critical dimensions:
Data Source Reality Check
Understanding your current data landscape isn't optional. This means cataloging:
- System inventory: What applications, databases, and sources exist today?
- Data quality baseline: What's the actual state of data completeness, accuracy, and consistency?
- Access patterns: How do systems currently integrate? What's technically feasible versus theoretically possible?
- Volume and velocity: What are the real numbers, not projected growth fantasies?
Audience and Use Case Mapping
Different consumers have fundamentally different needs:
- Business analysts need intuitive querying and reliable reporting with consistent definitions
- Data scientists require experimentation environments, feature engineering capabilities, and model deployment paths
- Operational systems demand real-time or near-real-time data with strong consistency guarantees
- Executive leadership wants trusted metrics, trend analysis, and strategic insights
- Regulatory and compliance teams need audit trails, data lineage, and security controls
Technical Requirements That Actually Matter
Beyond the obvious (volume, velocity, variety), critical technical requirements include:
Transaction integrity: Do you need ACID compliance? Are you processing financial transactions, inventory updates, or customer records where consistency matters more than speed?
Latency expectations: Is this truly real-time (milliseconds), near-real-time (seconds), or is batch processing (hours) actually acceptable?
Query patterns: Are you running ad-hoc exploration, predefined reports, complex analytical queries, or operational lookups?
Data freshness: What's the actual business value of minute-old versus hour-old versus day-old data?
Organizational Constraints
These factors determine success as much as technical capabilities:
- Team skills and learning capacity: Can your team absorb new technologies? What's the training timeline and cost?
- Time-to-value pressure: Do you need quick wins, or can you invest in long-term platforms?
- Budget realities: Not just licensing, but total cost of ownership, including support, training, and ongoing optimization
- Existing commitments: What cloud relationships or vendor partnerships already exist?
The Cost of Getting It Wrong: Two Real Scenarios
Case Study 1: The Premature Data Lake
A professional services organization embarked on a cloud migration, moving from on-premises SQL Server to AWS. Excited by the flexibility of S3 and the power of Spark, the team designed an elaborate data lake architecture. They would build raw, curated, and consumption layers. Everything would be configurable, scalable, and "future-proof."
Eighteen months and $2.3M later, the platform still couldn't reliably answer basic questions like "What was our revenue last quarter by practice area?"
The missed requirements:
- The organization's core data came from an ERP system with complex transactional relationships
- Most analytical queries required dimensional modeling with SCD Type 2 tracking to understand changes over time
- The finance team needed ACID guarantees—they couldn't tolerate eventual consistency when reconciling revenue
- The team lacked Spark expertise and underestimated the operational complexity of maintaining a custom platform
What they actually needed: A managed data warehouse (Snowflake or Azure Synapse) with strong ACID support and built-in SCD handling. They could have been operational in 12 weeks instead of 78.
The platform was eventually refactored, but not before significant organizational trust was damaged and opportunities were missed.
Case Study 2: Over-Engineering for Tomorrow
A mid-sized healthcare company decided to "invest in the future" by implementing Databricks across their entire analytics stack. The vision was compelling: machine learning, advanced analytics, unified data and AI platform.
The reality was different:
- 90% of use cases were standard BI reports and operational dashboards
- Their data science team consisted of one person doing basic predictive models
- Analysts spent months learning Spark and Scala when they were proficient in SQL
- The platform's flexibility became a burden—every team built different patterns, creating chaos instead of consistency
The missed requirements:
- Current state: overwhelmingly SQL-based reporting with occasional Python analysis
- Team composition: strong SQL skills, limited programming expertise
- Growth trajectory: ML was aspirational, not planned with budget or headcount
- Success criteria: reliable reports, fast query performance, analyst self-service
What they actually needed: Snowflake for core analytics with a small Databricks cluster for the one data scientist. This would have cost 60% less and delivered 3x faster time-to-value.
Both organizations made the classic mistake: they selected architecture before understanding requirements.
A Requirements-Driven Decision Framework
Here's a practical framework for making platform decisions based on actual needs rather than vendor promises or technology excitement.
Step 1: Map Your Data Workloads
Create a clear picture of what you're actually building:
| Workload Type | % of Total | Latency Need | User Type | Complexity |
|---|---|---|---|---|
| Standard BI Reports | 60% | Hours | Business Analysts | Low-Medium |
| Ad-Hoc Analysis | 20% | Minutes | Data Analysts | Medium |
| ML Model Training | 10% | Days | Data Scientists | High |
| Operational Dashboards | 8% | Seconds | Operations | Medium |
| Real-Time Alerts | 2% | Milliseconds | Automated Systems | High |
This simple exercise reveals what actually matters. In this example, 80% of workloads are traditional analytics, suggesting that a conventional warehouse might be the proper foundation.
Step 2: Assess Technical Requirements by Platform Strength
Different platforms excel at different capabilities:
| Requirement | Snowflake | Databricks | AWS (Custom) | Azure Synapse |
|---|---|---|---|---|
| ACID Transactions | Excellent | Good | Configurable | Excellent |
| SQL Performance | Excellent | Good | Configurable | Excellent |
| Spark Workloads | Limited | Excellent | Native | Good |
| ML/AI Integration | Growing | Excellent | Configurable | Good |
| Structured Data | Excellent | Good | Configurable | Excellent |
| Unstructured Data | Good | Excellent | Excellent | Good |
| Real-Time Streaming | Good | Excellent | Excellent | Good |
| Ease of Use (SQL users) | Excellent | Medium | Low | Good |
| Time to Production | Fast | Medium | Slow | Medium |
| Total Cost (typical analytics) | Medium | Medium-High | Variable | Medium |
Key insight: If 80% of your needs fall into SQL-heavy, structured analytics with ACID requirements, Snowflake or Synapse are natural fits. If you're doing heavy ML and unstructured processing, Databricks or custom AWS makes sense.
Step 3: Apply the Decision Matrix
Use this logic tree to guide platform selection:
Primary Question: What's your dominant workload pattern?
→ Primarily structured, SQL-based analytics with BI focus
- Team has strong SQL skills → Snowflake or Azure Synapse
- Need tight Azure integration → Azure Synapse
- Multi-cloud or cloud-agnostic → Snowflake
→ Heavy machine learning, data science, unstructured data
- Team has strong Python/Spark skills → Databricks
- Need unified data+AI platform → Databricks
- ML is exploratory/small-scale → Snowflake + separate ML tools
→ Real-time streaming, IoT, event processing
- High volume, complex transformations → AWS Kinesis/EMR or Azure Stream Analytics
- Simpler streaming needs → Databricks Structured Streaming
- Streaming + strong analytics → Databricks or Azure Synapse
→ Unique requirements, need ultimate flexibility
- Large engineering team, strong DevOps → Custom AWS/Azure
- Cost optimization critical, predictable workload → Custom AWS/Azure
- Most other scenarios → Managed platform recommended
Secondary Question: Do you need ACID transactions?
If YES and critical:
- Snowflake (best-in-class ACID for analytics)
- Azure Synapse (strong ACID support)
- Custom with PostgreSQL/SQL Server (if building)
If NO or eventual consistency acceptable:
- Databricks (sufficient for most ML/analytics)
- AWS Glue/Athena (for query-on-demand)
- Azure Data Lake (for flexibility)
Step 4: Factor in Organizational Reality
Technical fit is only half the equation:
Team Capability Assessment:
- Current skill set: SQL-heavy teams struggle with Spark-first platforms
- Learning capacity: Can you invest 3-6 months in training?
- Support needs: Do you need extensive vendor support or have strong internal expertise?
Timeline and Budget:
- Time-to-value pressure → Managed platforms (Snowflake, Synapse)
- Long-term flexibility → Custom builds on AWS/Azure (if you have the team)
- Budget constraints → Consider total cost: licensing + personnel + training + support
Strategic Alignment:
- Existing cloud relationships (AWS, Azure, GCP)
- Enterprise agreements and discounts
- Long-term cloud strategy
Platform Selection as an Outcome of Requirements
Let's walk through three realistic scenarios showing how requirements drive platform choice:
Scenario A: Financial Services Transaction Platform
Requirements:
- Source: Core banking system with 100M daily transactions
- Use cases: Regulatory reporting, fraud detection, customer analytics
- Key need: ACID compliance for financial accuracy
- Latency: Batch overnight for reports, near-real-time for fraud
- Team: Strong SQL, limited programming
- Timeline: 6 months to production
Decision Outcome: Azure Synapse
- Native ACID support for transaction integrity
- Strong SQL performance for reporting
- Azure integration for existing banking infrastructure
- Dedicated SQL pools for regulated workloads
- Familiar SQL interface for the team
- Complemented by Azure Stream Analytics for real-time fraud detection
Scenario B: E-Commerce Analytics and ML
Requirements:
- Sources: Web events (clickstream), product catalog, order systems
- Use cases: Customer behavior analysis, recommendation engine, inventory optimization
- Key need: ML model development and deployment
- Latency: Real-time recommendations, daily analytics
- Team: Data scientists (Python/Spark), analysts (SQL)
- Timeline: MVP in 3 months, full platform in 9 months
Decision Outcome: Databricks + Snowflake Hybrid
- Databricks for ML pipeline, feature engineering, real-time recommendations
- Snowflake for structured analytics and BI reporting
- Delta Lake for unified data layer
- Databricks ML for model management
- Snowflake for analyst self-service
- Best-of-breed for distinct workload patterns
Scenario C: Healthcare Data Integration
Requirements:
- Sources: 15+ legacy systems (EMR, billing, labs, pharmacy)
- Use cases: Population health analytics, regulatory reporting, research
- Key need: Data governance, security, lineage tracking
- Latency: Daily refresh acceptable
- Team: Small team, need vendor support
- Timeline: Quick wins in 3 months, phased rollout
Decision Outcome: Snowflake
- Quick time-to-value with minimal setup
- Strong security and governance features (HIPAA-ready)
- Easy integration with existing BI tools
- Managed service reduces operational burden
- Clear data lineage and audit trails
- Scales as organization grows
Building vs. Buying: The Custom Platform Question
A critical branch in the decision tree is whether to build custom on AWS/Azure or adopt a managed platform. Here's when each makes sense:
When Custom Makes Sense:
- You have a large, experienced engineering team comfortable with infrastructure-as-code, Terraform, and complex system design
- Your requirements are genuinely unique and no managed platform fits
- You have predictable, high-volume workloads where optimization can yield significant cost savings
- You need deep customization in processing logic, security, or integration
- Your organization has strong DevOps maturity
When Managed Platforms Win:
- You want fast time-to-value (weeks, not months)
- Your team is small or SQL-focused
- You need enterprise support and vendor partnership
- Your workloads are typical analytics patterns
- You want to focus on insights, not infrastructure
- You lack specialized platform engineering expertise
The hidden cost of "flexibility": Custom platforms require 2-4 full-time engineers for ongoing maintenance, optimization, and support. This often exceeds the cost savings from platform flexibility.
The maturity advantage: Managed platforms (Snowflake, Databricks, Synapse) benefit from thousands of customer workloads, continuous optimization, and enterprise-grade security. Unless you have an exceptional team and clear ROI, this maturity is hard to replicate.
Putting Requirements First: A Practical Approach
Here's how to implement this framework in your organization:
1. Conduct True Discovery (2-4 Weeks)
Workshops with stakeholders:
- Map current pain points, not future dreams
- Identify success criteria: what does "working" look like?
- Document decision-making timelines and urgency
Technical assessment:
- Audit existing data sources and quality
- Catalog current analytical patterns and queries
- Measure actual volumes, not theoretical growth
- Assess team skills honestly
Create a requirements matrix:
| Requirement | Priority | Current Gap | Technical Impact | Organizational Impact |
|---|---|---|---|---|
| Daily revenue reporting | High | 3-day lag | Need ACID, dimensional model | Trust in finance data |
| Customer segmentation | Medium | Manual Excel process | Need analytics SQL platform | Analyst productivity |
| Fraud detection | High | Not available | Need real-time scoring | Risk reduction |
2. Model Platform Options Against Requirements (1-2 Weeks)
For each viable platform, score against your requirements:
| Requirement | Weight | Snowflake Score | Databricks Score | Custom AWS Score |
|---|---|---|---|---|
| ACID for finance | 20% | 10/10 | 7/10 | 8/10 |
| ML capabilities | 15% | 6/10 | 10/10 | 9/10 |
| SQL performance | 25% | 10/10 | 7/10 | 7/10 |
| Time to value | 20% | 9/10 | 7/10 | 3/10 |
| Team fit | 20% | 9/10 | 6/10 | 4/10 |
| Weighted Total | 8.8 | 7.3 | 6.0 |
3. Present Options Tied to Requirements
When presenting to leadership, structure the conversation around needs, not tools:
"Based on our requirements analysis, we identified three critical needs:
- ACID-compliant financial reporting (addresses trust issue in finance data)
- Fast time-to-value (we need results in Q2, not Q4)
- Team skill alignment (our strength is SQL, not Spark engineering)
Given these requirements, we recommend Snowflake because:
- Delivers strongest ACID guarantees for financial accuracy
- Can be operational in 8-12 weeks with our current team
- Matches our SQL skillset, minimizing training costs
- Provides clear growth path as needs evolve
Alternative considered: Databricks
- Strengths: Superior ML capabilities, better for unstructured data
- Trade-offs: Longer implementation, team training needs, ML capabilities exceed current requirements
- Recommendation: Revisit in 18 months when the ML roadmap is clearer
Alternative considered: Custom AWS
- Strengths: Ultimate flexibility, potential long-term cost optimization
- Trade-offs: 12-18 month timeline, requires 3-4 additional engineers, operational complexity
- Recommendation: Only if requirements prove unique and managed platforms insufficient"
This approach shows strategic thinking, not just technology preference.
Conclusion: Strategy Before Selection
The pattern is clear across industries and company sizes: organizations that select platforms before understanding requirements face costly refactoring, missed opportunities, and organizational frustration.
The most successful data platform implementations follow a different path:
- Understand your data landscape with brutal honesty about current state
- Map actual use cases and the people who need them
- Define technical requirements that matter (ACID needs, latency, query patterns)
- Assess organizational readiness (skills, timeline, support needs)
- Select platforms that match requirements, not theoretical capabilities
Platform selection isn't strategy—it's the outcome of strategy.
The question isn't "Should we use Snowflake or Databricks or build on AWS?" The question is "What are we trying to accomplish, and which platform best enables that outcome given our constraints?"
Start with "what" and "why" before you jump to "how" and "where." Your architecture decisions should be answers to requirements, not the starting point of discovery.
The data platforms are powerful, mature, and capable. But they're tools, not solutions. The solution is understanding what you need to build, who needs to use it, and how it creates value. Once you have that clarity, platform selection becomes straightforward.
About the Author: Tony Henein is a data engineering leader with extensive experience building enterprise data platforms across financial services and professional services. He specializes in helping organizations navigate complex technology decisions by aligning architecture with business requirements and organizational capabilities.
Want to discuss your data platform strategy? Connect with me on LinkedIn or reach out through Acacia Tech Group.