Apache Spark

Config Commands

Configure Spark applications for performance and reliability. Set memory, cores, serialization, and shuffle parameters via command line or configuration files.

7 commands

Pro Tips

Set 'spark.serializer=org.apache.spark.serializer.KryoSerializer' for better performance.

Use 'spark.memory.fraction=0.6' to balance execution and storage memory.

Configure 'spark.sql.adaptive.enabled=true' for automatic query optimization (Spark 3+).

Common Mistakes

spark-defaults.conf changes affect all applications - use --conf for job-specific settings.

Incorrect memory configuration is the most common cause of Spark job failures.

Commands

Set config via CLI

$ spark-submit --conf spark.executor.memory=4g --conf spark.executor.cores=2 app.py

Pass configuration options via command line.

Use properties file

$ spark-submit --properties-file spark-defaults.conf app.py

Load configuration from a properties file.

Enable Kryo serializer

$ spark-submit --conf spark.serializer=org.apache.spark.serializer.KryoSerializer app.py

Use faster Kryo serialization instead of Java.

Configure memory fraction

$ spark-submit --conf spark.memory.fraction=0.8 --conf spark.memory.storageFraction=0.3 app.py

Tune execution vs storage memory balance.

Enable speculation

$ spark-submit --conf spark.speculation=true app.py

Enable speculative execution for straggler mitigation.

Set driver memory

$ spark-submit --driver-memory 8g --driver-cores 4 app.py

Configure driver resource allocation.

Configure broadcast threshold

$ spark-submit --conf spark.sql.autoBroadcastJoinThreshold=100m app.py

Set threshold for automatic broadcast joins.