Apache Spark

Submit Commands

Submit Spark applications to clusters. Configure deploy modes, resource allocation, dependencies, and cluster-specific options for optimal job execution.

9 commands

Pro Tips

Use '--deploy-mode cluster' for production to run driver on cluster and avoid network bottlenecks.

Set '--packages' for Maven coordinates to automatically download dependencies.

Use '--conf spark.dynamicAllocation.enabled=true' to auto-scale executors based on workload.

Common Mistakes

In client mode, driver runs locally - ensure network connectivity to cluster nodes.

Don't set executor memory higher than available node memory minus overhead.

Commands

Submit Python application

$ spark-submit app.py

Submit a PySpark application to the cluster.

Submit with master

$ spark-submit --master spark://host:7077 app.py

Submit to a specific Spark standalone master.

Submit to YARN cluster

$ spark-submit --master yarn --deploy-mode cluster app.py

Submit to YARN with driver running on cluster.

Configure executor resources

$ spark-submit --executor-memory 4g --executor-cores 2 --num-executors 10 app.py

Set executor memory, cores, and count.

Submit JAR application

$ spark-submit --class com.example.Main app.jar arg1 arg2

Submit Scala/Java application with main class.

Add dependencies

$ spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.0 app.py

Include Maven packages as dependencies.

Add Python files

$ spark-submit --py-files utils.zip,helpers.py app.py

Include additional Python files or archives.

Enable dynamic allocation

$ spark-submit --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.minExecutors=2 app.py

Auto-scale executors based on workload.

Submit to Kubernetes

$ spark-submit --master k8s://https://k8s-api:443 --deploy-mode cluster --conf spark.kubernetes.container.image=spark:3.5 app.py

Submit to Kubernetes cluster.