Use this checklist during design reviews and before promoting a Spark workload into production.

Data and storage

  • Input format supports efficient analytical reads
  • Partitioning strategy matches common access patterns
  • Schema evolution is defined
  • Small-file growth is monitored
  • Output ownership and retention are documented

Execution

  • Physical plan has been inspected
  • Shuffle-heavy stages are understood
  • Join strategy is appropriate for data size
  • Data skew has been tested
  • Executor memory and cores are deliberately sized
  • Spill and garbage-collection behavior are observable

PySpark

  • Built-in Spark functions are preferred where possible
  • Python UDF usage is justified and measured
  • Serialization overhead is understood
  • Dependency packaging is reproducible

Streaming

  • Checkpoint location is durable
  • Replay behavior has been tested
  • State growth is bounded and monitored
  • Event-time / watermark behavior is explicit
  • Sink failure and duplicate handling are understood

Operations

  • Driver and executor logs are collected
  • Job metrics and failure alerts exist
  • Retries do not hide deterministic failures
  • Data-quality checks run before publishing outputs
  • Runbooks exist for common failure scenarios
  • Cost / capacity is reviewed over representative workloads

Useful diagnostics

df.explain("formatted")
print(df.rdd.getNumPartitions())

Review the Spark UI for stage duration, shuffle read/write, spill, task skew and executor resource pressure.