Apache Spark

Shell Commands

Use interactive shells for Spark development. Access spark-shell (Scala), pyspark (Python), and sparkR for exploratory data analysis and prototyping.

7 commands

Pro Tips

Use ':paste' mode in spark-shell for multi-line code blocks.

Set 'PYSPARK_DRIVER_PYTHON=jupyter' to launch PySpark with Jupyter notebook.

SparkContext is pre-configured as 'sc' and SparkSession as 'spark' in shells.

Common Mistakes

Shell sessions consume cluster resources - remember to exit when done.

Large collect() operations in shell can crash driver with OutOfMemoryError.

Commands

Start Scala shell

$ spark-shell

Launch interactive Scala REPL with Spark context.

Start PySpark shell

$ pyspark

Launch interactive Python shell with Spark context.

PySpark with Jupyter

$ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=notebook pyspark

Launch PySpark with Jupyter notebook interface.

Shell with custom master

$ spark-shell --master spark://host:7077

Connect shell to specific Spark master.

Shell with packages

$ pyspark --packages org.apache.spark:spark-avro_2.12:3.5.0

Start shell with additional Maven dependencies.

Shell with memory config

$ spark-shell --driver-memory 4g --executor-memory 8g

Start shell with custom memory settings.

Start SparkR shell

$ sparkR

Launch interactive R shell with Spark context.