Run Commands in Parallel with xargs and GNU Parallel

Advanced15 min

Speed up batch operations by executing commands in parallel using xargs -P, GNU parallel, and background jobs with wait.

Prerequisites

-Bash 4.0+
-GNU coreutils (xargs)
-GNU parallel (optional, for advanced usage)

Steps

Run commands in parallel with xargs

Use xargs -P to run multiple processes simultaneously.

$ find . -name '*.png' -print0 | xargs -0 -P 4 -I {} optipng -o2 {}

-P 4 runs 4 processes in parallel. -print0 and -0 handle filenames with spaces safely.

Process a list of URLs in parallel

Download or check multiple URLs concurrently using xargs.

$ cat urls.txt | xargs -P 8 -I {} curl -s -o /dev/null -w '{} %{http_code}\n' {}

Use GNU parallel for advanced parallel execution

GNU parallel provides job logging, progress bars, and automatic load balancing.

$ find . -name '*.csv' | parallel -j 4 --bar 'gzip {}'

Install with: brew install parallel (macOS) or apt install parallel (Ubuntu).

Use background jobs with wait

Run commands as background jobs and wait for all to complete.

$ for i in 1 2 3 4; do (echo "Job $i starting" && sleep $((RANDOM % 5)) && echo "Job $i done") & done wait echo "All jobs complete"

Without 'wait', the script may exit before background jobs finish. Always call wait before using the results.

Limit concurrency with a semaphore pattern

Control the maximum number of concurrent background jobs.

$ MAX_JOBS=4 for file in *.log; do while [[ $(jobs -r | wc -l) -ge $MAX_JOBS ]]; do sleep 0.1; done process_file "$file" & done wait

Full Script

full-script.sh

#!/usr/bin/env bash
set -euo pipefail

# Parallel Execution Examples
MAX_PARALLEL=${1:-4}
ACTION=${2:-echo}

log() { echo "[$(date +'%H:%M:%S')] $*"; }

# Method 1: xargs -P
log "=== xargs parallel (max $MAX_PARALLEL) ==="
find . -maxdepth 1 -name '*.txt' -print0 | xargs -0 -P "$MAX_PARALLEL" -I {} sh -c '
  echo "Processing: {}"
  wc -l "{}"
'

# Method 2: Background jobs with wait
log "=== Background jobs ==="
PIDS=()
for i in $(seq 1 "$MAX_PARALLEL"); do
  (
    log "Job $i started"
    sleep 1
    log "Job $i completed"
  ) &
  PIDS+=($!)
done

# Wait and check exit codes
FAILED=0
for pid in "${PIDS[@]}"; do
  wait "$pid" || ((FAILED++))
done

log "All jobs complete. Failures: $FAILED"
[[ $FAILED -gt 0 ]] && exit 1 || exit 0

FAQ

Discussion

Loading comments...

#!/usr/bin/env bash set -euo pipefail # Parallel Execution Examples MAX_PARALLEL=${1:-4} ACTION=${2:-echo} log() { echo "[$(date +'%H:%M:%S')] $*"; } # Method 1: xargs -P log "=== xargs parallel (max $MAX_PARALLEL) ===" find . -maxdepth 1 -name '*.txt' -print0 | xargs -0 -P "$MAX_PARALLEL" -I {} sh -c ' echo "Processing: {}" wc -l "{}" ' # Method 2: Background jobs with wait log "=== Background jobs ===" PIDS=() for i in $(seq 1 "$MAX_PARALLEL"); do ( log "Job $i started" sleep 1 log "Job $i completed" ) & PIDS+=($!) done # Wait and check exit codes FAILED=0 for pid in "${PIDS[@]}"; do wait "$pid" || ((FAILED++)) done log "All jobs complete. Failures: $FAILED" [[ $FAILED -gt 0 ]] && exit 1 || exit 0