Use Private Networks with your Apache Spark™ cluster

Reviewed on September 10, 2026

Private Networks allow your Clusters for Apache Spark™ cluster to communicate in an isolated and secure network without needing to be connected to the public internet.

At the moment, Apache Spark™ clusters can only be attached to a Private Network during their creation, and cannot be detached and reattached to another Private Network afterward.

For full information about Scaleway Private Networks and VPC, see our dedicated documentation and best practices guide.

Note

You can attach this product to a Private Network, but it currently does not support VPC routing. It is therefore not compatible with VPC Peering.

Before you start

To complete the actions presented below, you must have:

A Scaleway account logged into the console (External link)
Owner status or IAM permissions allowing you to perform actions in the intended Organization
Created a Private Network
Created an Ubuntu Instance attached to a Private Network

Set up your Instance with Python 3.13 and Spark tools

Connect to your Instance via SSH.
Run the following commands from the shell of your Instance to install the required dependencies:
```
sudo apt update
sudo apt install -y \
  build-essential zlib1g-dev libssl-dev libbz2-dev libsqlite3-dev \
  libreadline-dev libncurses-dev liblzma-dev libffi-dev \
  openjdk-21-jre-headless
```
Note

Pyenv requires some dev packages for building a python version.

Spark 4.0 runs on Java 17/21 (a too recent version of the Java runtime might not work).
Run the following command to install pyenv:
```
curl https://pyenv.run | bash
```

Run the following commands to add pyenv to your Bash configuration:

echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc

Run the following command to reload your shell:
```
exec $SHELL
```
Run the following command to install Python 3.13:
```
# Download and install the latest Python 3.13
pyenv install 3.13
# Sets the default python version to 3.13
# If you prefer not to change this setting globally, use a pyenv local config.
pyenv global 3.13
```
Note
Your Instance's Python version must be 3.13. If you encounter an error due to a mismatch between the worker and driver Python versions, run the following command to display minor versions, then reinstall using the exact one:
pyenv install -l | grep 3.13

Run the following command to install Apache Spark™:

cd ~
wget https://archive.apache.org/dist/spark/spark-4.0.0/spark-4.0.0-bin-hadoop3.tgz
sudo mkdir -p /opt/spark
sudo tar -xzf spark-4.0.0-bin-hadoop3.tgz -C /opt/spark --strip-components=1

(Optional) Run the following command to install support for Object Storage/S3:

cd /opt/spark/jars && curl -fSL --remote-name-all                                           \
    https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.4.1/hadoop-aws-3.4.1.jar  \
    https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.24.6/bundle-2.24.6.jar   \
    https://repo1.maven.org/maven2/org/wildfly/openssl/wildfly-openssl/1.1.3.Final/wildfly-openssl-1.1.3.Final.jar

Run the following commands to add Apache Spark™ to your Bash configuration:

echo 'export SPARK_HOME=/opt/spark' >> ~/.bashrc
echo 'export PATH="$SPARK_HOME/bin:$PATH"' >> ~/.bashrc

Run the following command to reload your shell:
```
exec $SHELL
```

Run Spark examples on your cluster with spark-submit

Run a Python application

Connect to your Instance via SSH.
Verify that you are using Python 3.13.x:
```
python3 --version
```
Run spark-submit with the following command to calculate pi for 1000 iterations. Do not forget to replace the placeholders with the appropriate values.
```
export SPARK_LOCAL_IP=<INSTANCE_PN_IP>
spark-submit \
--master spark://<SPARK_MASTER_ENDPOINT>:7077 \
--deploy-mode client \
$SPARK_HOME/examples/src/main/python/pi.py 1000
```
Note

<SPARK_MASTER_ENDPOINT> can be found in the Overview tab of your cluster, under Private endpoint in the Network section.

<INSTANCE_PN_IP> can be found in the Private Networks tab of your Instance. Make sure to only copy the IP, and not the /22 part.
Access the Apache Spark™ UI of your cluster. The list of completed applications displays. From here, you can inspect the jobs previously started using spark-submit.

Run a Jvm application (Java or Scala)

Connect to your Instance via SSH.

Run the SparkPi application in client mode with the following command:

export SPARK_LOCAL_IP=<INSTANCE_PN_IP>
spark-submit \
--master spark://<SPARK_MASTER_ENDPOINT>:7077 \
--deploy-mode client \
--class org.apache.spark.examples.SparkPi \
$SPARK_HOME/examples/jars/spark-examples_2.13-4.0.0.jar 1000

Run the SparkPi application in cluster mode with the following command:
```
UNSET SPARK_LOCAL_IP
spark-submit  \
--master spark://<SPARK_MASTER_ENDPOINT>:7077 \
--deploy-mode cluster \
--conf spark.driver.port=37000 --conf spark.blockManager.port=38000 \
--class org.apache.spark.examples.SparkPi \
$SPARK_HOME/examples/jars/spark-examples_2.13-4.0.0.jar 1000
```
Note

spark.driver.port and spark.blockManager.port need to be in the 37000-38999 port range since only these ports are allowed in Spark clusters.

The spark-submit command will exit as soon as the driver has been started on the cluster. Check out the Spark UI to get the Pi result.

Use Object storage (S3)

Create a spark-defaults.conf file

To avoid passing all options to spark-submit, let's create a /opt/spark/conf/spark-defaults.conf file with the following content:

spark.master:                                 spark://<SPARK_MASTER_ENDPOINT>:7077
spark.driver.port:                            37000
spark.blockManager.port:                      38000
spark.hadoop.fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
spark.hadoop.fs.s3a.endpoint:                 https://s3.fr-par.scw.cloud
spark.hadoop.fs.s3a.endpoint.region:          fr-par

Remember to replace the following:

<SPARK_MASTER_ENDPOINT>: You can find the value of <SPARK_MASTER_ENDPOINT> in the Overview tab of your cluster, under Private endpoint in the Network section.
S3 endpoint and region: Note that these values must match the region of your Spark cluster if running spark-submit in cluster mode.

Run a Jvm application stored in S3

Upload your jar file to Object Storage and make it public. For the sake of this example, let's assume that we uploaded a file called spark-examples_2.13-4.0.0.jar.
Connect to your Instance via SSH.

Run your application in client mode with the following command:

export SPARK_LOCAL_IP=<INSTANCE_PN_IP>
spark-submit \
--deploy-mode client \
--class org.apache.spark.examples.SparkPi \
s3a://<bucket_name>/spark-examples_2.13-4.0.0.jar 1000

Remember to replace the placeholder values and the name of the jar file with your actual values.

Run your application application in cluster mode with the following command:

UNSET SPARK_LOCAL_IP
spark-submit  \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
s3a://<bucket_name>/spark-examples_2.13-4.0.0.jar 1000

Run a Jvm application stored in a private S3

Running a Jvm application stored in a private S3 is only possible in client mode.

Upload your jar file to Object Storage, make it private, and generate an IAM API key with at least the ObjectStorageReadOnly permission.

Update your /opt/spark/conf/spark-defaults.conf file with the following content:

spark.master:                                 spark://<SPARK_MASTER_ENDPOINT>:7077
spark.hadoop.fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
spark.hadoop.fs.s3a.endpoint:                 https://s3.fr-par.scw.cloud
spark.hadoop.fs.s3a.endpoint.region:          fr-par
spark.hadoop.fs.s3a.access.key:               <SCW_ACCESS_KEY>
spark.hadoop.fs.s3a.secret.key:               <SCW_SECRET_KEY>

Remember to replace the placeholder values and the region and endpoint information with your actual values.

Run your application with the following command:

export SPARK_LOCAL_IP=<INSTANCE_PN_IP>
spark-submit \
--deploy-mode client \
--class org.apache.spark.examples.SparkPi \
s3a://<bucket_name>/spark-examples_2.13-4.0.0.jar 1000

Remember to replace the placeholder values and the name of the jar file with your actual values.