Apache Spark

Cluster Commands

Deploy Spark on different cluster managers. Configure standalone clusters, YARN queues, or Kubernetes pods for scalable distributed computing.

9 commands

Pro Tips

Use '--master yarn --deploy-mode cluster' for production YARN deployments.

On Kubernetes, set 'spark.kubernetes.container.image' to your custom Spark image.

Enable '--supervise' in standalone mode to auto-restart failed drivers.

Common Mistakes

YARN queue limits can cause applications to wait indefinitely - monitor queue capacity.

Kubernetes deployments require proper RBAC permissions for the Spark service account.

Commands

Start standalone master

$ $SPARK_HOME/sbin/start-master.sh

Start Spark standalone cluster master.

Start worker

$ $SPARK_HOME/sbin/start-worker.sh spark://master:7077

Start worker and connect to master.

Start all workers

$ $SPARK_HOME/sbin/start-workers.sh

Start workers on all machines in conf/workers file.

Stop cluster

$ $SPARK_HOME/sbin/stop-all.sh

Stop master and all workers.

Submit with supervise

$ spark-submit --deploy-mode cluster --supervise --master spark://host:7077 app.py

Enable automatic driver restart on failure.

Kill application

$ spark-submit --kill driver-20240101120000-0001 --master spark://host:7077

Kill a running application on standalone cluster.

YARN queue submission

$ spark-submit --master yarn --queue production --deploy-mode cluster app.py

Submit to specific YARN queue.

K8s with service account

$ spark-submit --master k8s://https://k8s-api:443 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark app.py

Submit to Kubernetes with service account.

Check cluster status

$ curl http://spark-master:8080/json/

Query Spark master REST API for cluster status.