INGEST → PROCESS → STREAM → ORCHESTRATE → SERVE

Data Engineering & Big Data

Understand distributed data systems and build practical batch, streaming and lakehouse pipelines with production-oriented performance and operability.

Typical duration3–8 days; modular Spark, Kafka or platform-specific tracks available

DeliveryOnline · On-site · Hybrid · Data-platform lab

Practical work5 representative labs

Outcomes

Explain distributed storage and compute trade-offs

Build and tune Spark / PySpark workloads

Design batch and streaming pipelines

Understand lakehouse, orchestration and distributed SQL patterns

Audience & prerequisites

Who it is for— Data engineers— Platform engineers supporting data workloads— Developers moving into distributed data— Architects evaluating modern data platforms

Prerequisites— Linux fundamentals— Basic Python and SQL— Distributed-systems concepts are introduced as needed

Representative curriculum

Training topics, organized as engineering modules.

The exact sequence is adjusted to the audience. Foundation topics can be compressed for experienced teams; architecture and troubleshooting can be expanded for advanced programs.

Big Data & Distributed Systems

Build the mental model behind modern data platforms.

Scale-up vs scale-out
Partitioning and parallelism
Distributed storage
Failure and retries
Batch vs streaming

Apache Spark

Learn Spark from execution model to production tuning.

Driver, executors and cluster managers
DataFrames and Spark SQL
PySpark
Partitions and shuffles
Caching and joins
Performance diagnostics

Streaming & Kafka

Design event-driven data movement.

Kafka brokers, topics and partitions
Producers and consumers
Offsets and delivery semantics
Spark Structured Streaming
Reliability and back pressure

Distributed SQL & Serving

Query data across platforms efficiently.

Trino architecture
Connectors and catalogs
Query planning
Federated query
Performance and security

Lakehouse Architecture

Connect object storage, table formats and compute engines.

Data lakes vs warehouses vs lakehouse
Iceberg / Delta concepts
Schema evolution
Partition design
Spark / Trino interoperability

Orchestration & Operations

Make pipelines observable and repeatable.

Airflow concepts
Dependency management
Retries and idempotency
Data-quality checks
Monitoring and troubleshooting

Hands-on work

Labs are part of the learning path.

Exercises emphasize observation, implementation, failure and diagnosis rather than command copying.

Build and tune a PySpark batch pipeline

Inspect a Spark execution plan and shuffle

Process Kafka events using Structured Streaming

Query lakehouse data with Trino

Orchestrate a multi-stage pipeline

CONNECTED PATHS

Turn the learning path into production context.

Explore Knowledge Hub

ConsultingData Engineering

Data Engineering & Big Data

Architect, modernize and troubleshoot distributed data platforms spanning Apache Spark, Kafka, Trino, orchestration and lakehouse technologies.

Continue

Case StudyData Engineering

Recovering a slow Spark data pipeline

A representative performance-engineering scenario for a Spark/PySpark workload that becomes slower and less predictable as data volume and transformation complexity grow.

Continue

ArticleData Engineering