A modern data platform is not one product. It is a set of responsibilities: ingest data, store it durably, process it, coordinate workflows, serve queries, govern access and observe the whole path.
Applications / Databases / Events → Kafka / CDC / Batch ingestion → Object storage + table format → Spark / Flink → Trino / BI / ML / AI
Separate responsibilities
A clear architecture avoids asking every technology to solve every problem.
| Responsibility | Representative technologies |
|---|---|
| Event ingestion | Kafka |
| Batch / distributed compute | Apache Spark |
| Stream processing | Spark Structured Streaming, Apache Flink |
| Workflow orchestration | Apache Airflow |
| Durable analytical storage | object storage / HDFS |
| Table format | Apache Iceberg, Delta Lake |
| Distributed SQL | Trino |
| Observability | metrics, logs, traces, data-quality signals |
The exact stack should be selected from workload requirements rather than from a product checklist.
Batch and streaming together
Many organizations need both scheduled transformations and low-latency event processing. The important architectural decision is how those paths converge into governed datasets that downstream systems can trust.
A simple pattern is:
Batch sources → Spark ┐ ├→ governed lakehouse tables → Trino / BI / ML / AI Kafka events → Spark / Flink ┘
Spark
Spark is useful when distributed transformations, SQL analytics, machine-learning preparation or streaming pipelines need to operate over large datasets. Production design should include partitioning, shuffle behavior, resource management and workload isolation.
Kafka
Kafka provides a durable distributed event log. Architecture decisions include topic design, partition count, retention, producer acknowledgement, consumer groups, schema evolution and failure recovery.
Trino
Trino is a distributed SQL query engine that can query multiple data sources through connectors. It is useful when interactive SQL or federation is required without copying every source into a single database.
Lakehouse table formats
Open table formats such as Apache Iceberg can add snapshot-based tables, schema evolution and partition evolution on object storage. Delta Lake provides a related table-management model. The platform should standardize ownership, compaction, retention and metadata management rather than treating the table format as magic infrastructure.
Airflow and orchestration
Airflow coordinates workflows; it is not the data-processing engine itself. Use orchestration to express dependencies, retries, schedules and operational state while Spark, SQL engines or services perform the actual compute.
Production design questions
Before choosing a stack, answer:
- What data volumes and arrival rates are expected?
- Which workloads need seconds, minutes or hours of latency?
- What are the recovery and replay requirements?
- Where is the system of record?
- What schema and data-quality guarantees exist?
- Which workloads are interactive versus scheduled?
- How will access, lineage and retention be governed?
- How will engineers see failed pipelines and expensive queries?