Clusters for Apache Spark™

Download schema

Each cluster is composed of one or more dedicated compute nodes running Apache Spark™. It can optionally include a dedicated JupyterLab Notebook. These resources are fully manageable through this API.

GET

https://api.scaleway.com

/datalab/v1beta1/regions/{region}/cluster-versions

List the Apache Spark™ versions the product is compatible with.

path Parameters

region

string · enum · required

The region you want to target

Enum values:

fr-par

it-mil

query Parameters

page

integer · int32

The page number.

page_size

integer · uint32

The page size.

order_by

scaleway.datalab.v1beta1.ListClusterVersionsRequest.OrderBy · enum

The order by field.

Enum values:

name_asc

name_desc

Default: name_asc

Responses

200

scaleway.datalab.v1beta1.ListClusterVersionsResponse

scaleway.datalab.v1beta1.Cluster[]

The list of cluster versions.

total_count

integer · uint64

The total count of cluster versions.

GET

https://api.scaleway.com

/datalab/v1beta1/regions/{region}/datalabs

List information about cluster within a project or an organization.

path Parameters

region

string · enum · required

The region you want to target

Enum values:

fr-par

it-mil

query Parameters

organization_id

string

The unique identifier of the organization whose clusters you want to list.

project_id

string

The unique identifier of the project whose clusters you want to list.

name

string

The name of the cluster you want to list.

tags

string[]

The tags associated with the cluster you want to list.

page

integer · int32

The page number for pagination.

page_size

integer · uint32

The page size for pagination.

order_by

scaleway.datalab.v1beta1.ListDatalabsRequest.OrderBy · enum

The order by field, available options are name_asc, name_desc, created_at_asc, created_at_desc, updated_at_asc, updated_at_desc.

Enum values:

name_asc

name_desc

created_at_asc

created_at_desc

updated_at_asc

updated_at_desc

Default: name_asc

Responses

200

scaleway.datalab.v1beta1.ListDatalabsResponse

scaleway.datalab.v1beta1.Datalab[]

The list of clusters. This is a list composed of messages of type DataLab.

total_count

integer · uint64

The total count of clusters.

GET/datalab/v1beta1/regions/{region}/datalabs

curl -X GET \
  -H "X-Auth-Token: $SCW_SECRET_KEY" \
  "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs"

Example Responses

{
  "datalabs": [
    {
      "id": "6170692e-7363-616c-6577-61792e636f6d",
      "project_id": "6170692e-7363-616c-6577-61792e636f6d",
      "name": "name",
      "description": "description",
      "tags": [
        "string"
      ],
      "main": {
        "node_type": "node_type",
        "spark_ui_url": "spark_ui_url",
        "spark_master_url": "spark_master_url",
        "root_volume": {
          "type": "unknown_type",
          "size": 0
        }
      },
      "worker": {
        "node_type": "node_type",
        "node_count": 0,
        "root_volume": {
          "type": "unknown_type",
          "size": 0
        }
      },
      "status": "unknown_status",
      "created_at": "2022-03-22T12:34:56.123456Z",
      "updated_at": "2022-03-22T12:34:56.123456Z",
      "region": "region",
      "has_notebook": true,
      "notebook_url": "notebook_url",
      "spark_version": "spark_version",
      "total_storage": {
        "type": "unknown_type",
        "size": 0
      },
      "private_network_id": "6170692e-7363-616c-6577-61792e636f6d",
      "notebook_master_url": "notebook_master_url"
    }
  ],
  "total_count": 0
}

json

application/json

POST

https://api.scaleway.com

/datalab/v1beta1/regions/{region}/datalabs

Create a new cluster. In this call, one can personalize the node counts, add a notebook, choose the private network, define the persistent volume storage capacity.

path Parameters

region

string · enum · required

The region you want to target

Enum values:

fr-par

it-mil

Request Body

project_id

string

The unique identifier of the project where the cluster will be created.

name

string

The name of the cluster.

description

string

The description of the cluster.

tags

string[]

The tags of the cluster.

scaleway.datalab.v1beta1.CreateDatalabRequest.SparkMain

The cluster main node specification. It holds the parameters node_type which specifies the node type of the main node. See ListNodeTypes for available options. See ListNodeTypes for available options.

scaleway.datalab.v1beta1.CreateDatalabRequest.SparkWorker

The cluster worker node specification. It holds the parameters node_type which specifies the node type of the worker node and node_count for specifying the amount of nodes.

has_notebook

boolean

Select this option to include a notebook as part of the cluster.

spark_version

string

The version of Apache Spark™ running inside the cluster, available options can be viewed at ListClusterVersions.

scaleway.datalab.v1beta1.Volume

The maximum persistent volume storage that will be available during workload.

private_network_id

string

The unique identifier of the private network the cluster will be attached to.

Responses

200

scaleway.datalab.v1beta1.Datalab

id

string

The unique identifier of the cluster. (UUID format)

project_id

string

The unique identifier of the project where the cluster has been created. (UUID format)

name

string

The name of the cluster.

description

string

The description of the cluster.

tags

string[]

The tags of the cluster.

scaleway.datalab.v1beta1.Datalab.SparkMain

The Apache Spark™ Main node specification of cluster. It holds the parameters node_type, spark_ui_url (available to reach Apache Spark™ UI), spark_master_url (used to reach the cluster within a VPC), root_volume (size of the volume assigned to the cluster).

scaleway.datalab.v1beta1.Datalab.SparkWorker

The cluster worker nodes specification. It holds the parameters node_type, node_count, root_volume (size of the volume assigned to the cluster).

status

scaleway.datalab.v1beta1.Datalab.Status · enum

The status of the cluster. For a working cluster the status is marked as ready.

Enum values:

unknown_status

creating

updating

ready

error

deleting

locked

deleted

Default: unknown_status

created_at

string | null · date-time

The creation timestamp of the cluster. (RFC 3339 format)

updated_at

string | null · date-time

The last update date of the cluster. (RFC 3339 format)

region

string

The region of the cluster.

has_notebook

boolean

Whether a JupyterLab notebook is associated with the cluster or not.

notebook_url

string | null

The URL of the notebook if available.

spark_version

string

The version of Apache Spark™ running inside the cluster.

scaleway.datalab.v1beta1.Volume

The total persistent volume storage selected to run Apache Spark™.

private_network_id

string

The unique identifier of the private network to which the cluster is attached to. (UUID format)

notebook_master_url

string | null

The URL that is used to reach the cluster from the notebook when available. This URL cannot be used to reach the cluster from a server.

POST/datalab/v1beta1/regions/{region}/datalabs

curl -X POST \
  -H "X-Auth-Token: $SCW_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "string",
    "has_notebook": false,
    "name": "string",
    "private_network_id": "string",
    "project_id": "string",
    "spark_version": "string"
  }' \
  "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs"

Example Request Body

{
  "project_id": "project_id",
  "name": "name",
  "description": "description",
  "tags": [
    "string"
  ],
  "main": {
    "node_type": "node_type"
  },
  "worker": {
    "node_type": "node_type",
    "node_count": 0
  },
  "has_notebook": true,
  "spark_version": "spark_version",
  "total_storage": {
    "type": "unknown_type",
    "size": 0
  },
  "private_network_id": "private_network_id"
}

json

Example Responses

{
  "id": "6170692e-7363-616c-6577-61792e636f6d",
  "project_id": "6170692e-7363-616c-6577-61792e636f6d",
  "name": "name",
  "description": "description",
  "tags": [
    "string"
  ],
  "main": {
    "node_type": "node_type",
    "spark_ui_url": "spark_ui_url",
    "spark_master_url": "spark_master_url",
    "root_volume": {
      "type": "unknown_type",
      "size": 0
    }
  },
  "worker": {
    "node_type": "node_type",
    "node_count": 0,
    "root_volume": {
      "type": "unknown_type",
      "size": 0
    }
  },
  "status": "unknown_status",
  "created_at": "2022-03-22T12:34:56.123456Z",
  "updated_at": "2022-03-22T12:34:56.123456Z",
  "region": "region",
  "has_notebook": true,
  "notebook_url": "notebook_url",
  "spark_version": "spark_version",
  "total_storage": {
    "type": "unknown_type",
    "size": 0
  },
  "private_network_id": "6170692e-7363-616c-6577-61792e636f6d",
  "notebook_master_url": "notebook_master_url"
}

json

application/json

GET

https://api.scaleway.com

/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}

Retrieve information about a given cluster, specified by the region and datalab_id parameters. Its full details, including name, status, node counts, are returned in the response object.

path Parameters

region

string · enum · required

The region you want to target

Enum values:

fr-par

it-mil

datalab_id

string · required

The unique identifier of the cluster.

Responses

200

scaleway.datalab.v1beta1.Datalab

id

string

The unique identifier of the cluster. (UUID format)

project_id

string

The unique identifier of the project where the cluster has been created. (UUID format)

name

string

The name of the cluster.

description

string

The description of the cluster.

tags

string[]

The tags of the cluster.

scaleway.datalab.v1beta1.Datalab.SparkMain

scaleway.datalab.v1beta1.Datalab.SparkWorker

The cluster worker nodes specification. It holds the parameters node_type, node_count, root_volume (size of the volume assigned to the cluster).

status

scaleway.datalab.v1beta1.Datalab.Status · enum

The status of the cluster. For a working cluster the status is marked as ready.

Enum values:

unknown_status

creating

updating

ready

error

deleting

locked

deleted

Default: unknown_status

created_at

string | null · date-time

The creation timestamp of the cluster. (RFC 3339 format)

updated_at

string | null · date-time

The last update date of the cluster. (RFC 3339 format)

region

string

The region of the cluster.

has_notebook

boolean

Whether a JupyterLab notebook is associated with the cluster or not.

notebook_url

string | null

The URL of the notebook if available.

spark_version

string

The version of Apache Spark™ running inside the cluster.

scaleway.datalab.v1beta1.Volume

The total persistent volume storage selected to run Apache Spark™.

private_network_id

string

The unique identifier of the private network to which the cluster is attached to. (UUID format)

notebook_master_url

string | null

The URL that is used to reach the cluster from the notebook when available. This URL cannot be used to reach the cluster from a server.

GET/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}

curl -X GET \
  -H "X-Auth-Token: $SCW_SECRET_KEY" \
  "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}"

Example Responses

{
  "id": "6170692e-7363-616c-6577-61792e636f6d",
  "project_id": "6170692e-7363-616c-6577-61792e636f6d",
  "name": "name",
  "description": "description",
  "tags": [
    "string"
  ],
  "main": {
    "node_type": "node_type",
    "spark_ui_url": "spark_ui_url",
    "spark_master_url": "spark_master_url",
    "root_volume": {
      "type": "unknown_type",
      "size": 0
    }
  },
  "worker": {
    "node_type": "node_type",
    "node_count": 0,
    "root_volume": {
      "type": "unknown_type",
      "size": 0
    }
  },
  "status": "unknown_status",
  "created_at": "2022-03-22T12:34:56.123456Z",
  "updated_at": "2022-03-22T12:34:56.123456Z",
  "region": "region",
  "has_notebook": true,
  "notebook_url": "notebook_url",
  "spark_version": "spark_version",
  "total_storage": {
    "type": "unknown_type",
    "size": 0
  },
  "private_network_id": "6170692e-7363-616c-6577-61792e636f6d",
  "notebook_master_url": "notebook_master_url"
}

json

application/json

DELETE

https://api.scaleway.com

/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}

Delete a cluster based on its region and id.

path Parameters

region

string · enum · required

The region you want to target

Enum values:

fr-par

it-mil

datalab_id

string · required

The unique identifier of the cluster.

Responses

200

scaleway.datalab.v1beta1.Datalab

id

string

The unique identifier of the cluster. (UUID format)

project_id

string

The unique identifier of the project where the cluster has been created. (UUID format)

name

string

The name of the cluster.

description

string

The description of the cluster.

tags

string[]

The tags of the cluster.

scaleway.datalab.v1beta1.Datalab.SparkMain

scaleway.datalab.v1beta1.Datalab.SparkWorker

The cluster worker nodes specification. It holds the parameters node_type, node_count, root_volume (size of the volume assigned to the cluster).

status

scaleway.datalab.v1beta1.Datalab.Status · enum

The status of the cluster. For a working cluster the status is marked as ready.

Enum values:

unknown_status

creating

updating

ready

error

deleting

locked

deleted

Default: unknown_status

created_at

string | null · date-time

The creation timestamp of the cluster. (RFC 3339 format)

updated_at

string | null · date-time

The last update date of the cluster. (RFC 3339 format)

region

string

The region of the cluster.

has_notebook

boolean

Whether a JupyterLab notebook is associated with the cluster or not.

notebook_url

string | null

The URL of the notebook if available.

spark_version

string

The version of Apache Spark™ running inside the cluster.

scaleway.datalab.v1beta1.Volume

The total persistent volume storage selected to run Apache Spark™.

private_network_id

string

The unique identifier of the private network to which the cluster is attached to. (UUID format)

notebook_master_url

string | null

The URL that is used to reach the cluster from the notebook when available. This URL cannot be used to reach the cluster from a server.

DELETE/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}

curl -X DELETE \
  -H "X-Auth-Token: $SCW_SECRET_KEY" \
  "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}"

Example Responses

{
  "id": "6170692e-7363-616c-6577-61792e636f6d",
  "project_id": "6170692e-7363-616c-6577-61792e636f6d",
  "name": "name",
  "description": "description",
  "tags": [
    "string"
  ],
  "main": {
    "node_type": "node_type",
    "spark_ui_url": "spark_ui_url",
    "spark_master_url": "spark_master_url",
    "root_volume": {
      "type": "unknown_type",
      "size": 0
    }
  },
  "worker": {
    "node_type": "node_type",
    "node_count": 0,
    "root_volume": {
      "type": "unknown_type",
      "size": 0
    }
  },
  "status": "unknown_status",
  "created_at": "2022-03-22T12:34:56.123456Z",
  "updated_at": "2022-03-22T12:34:56.123456Z",
  "region": "region",
  "has_notebook": true,
  "notebook_url": "notebook_url",
  "spark_version": "spark_version",
  "total_storage": {
    "type": "unknown_type",
    "size": 0
  },
  "private_network_id": "6170692e-7363-616c-6577-61792e636f6d",
  "notebook_master_url": "notebook_master_url"
}

json

application/json

PATCH

https://api.scaleway.com

/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}

Update a cluster node counts. Allows for up- and downscaling on demand, depending on the expected workload.

path Parameters

region

string · enum · required

The region you want to target

Enum values:

fr-par

it-mil

datalab_id

string · required

The unique identifier of the cluster.

Request Body

name

string | null

The updated name of the cluster.

description

string | null

The updated description of the cluster.

tags

string[]

The updated tags of the cluster.

node_count

integer | null · uint32

The updated node count of the cluster. Scale up or down the number of worker nodes.

Responses

200

scaleway.datalab.v1beta1.Datalab

id

string

The unique identifier of the cluster. (UUID format)

project_id

string

The unique identifier of the project where the cluster has been created. (UUID format)

name

string

The name of the cluster.

description

string

The description of the cluster.

tags

string[]

The tags of the cluster.

scaleway.datalab.v1beta1.Datalab.SparkMain

scaleway.datalab.v1beta1.Datalab.SparkWorker

The cluster worker nodes specification. It holds the parameters node_type, node_count, root_volume (size of the volume assigned to the cluster).

status

scaleway.datalab.v1beta1.Datalab.Status · enum

The status of the cluster. For a working cluster the status is marked as ready.

Enum values:

unknown_status

creating

updating

ready

error

deleting

locked

deleted

Default: unknown_status

created_at

string | null · date-time

The creation timestamp of the cluster. (RFC 3339 format)

updated_at

string | null · date-time

The last update date of the cluster. (RFC 3339 format)

region

string

The region of the cluster.

has_notebook

boolean

Whether a JupyterLab notebook is associated with the cluster or not.

notebook_url

string | null

The URL of the notebook if available.

spark_version

string

The version of Apache Spark™ running inside the cluster.

scaleway.datalab.v1beta1.Volume

The total persistent volume storage selected to run Apache Spark™.

private_network_id

string

The unique identifier of the private network to which the cluster is attached to. (UUID format)

notebook_master_url

string | null

The URL that is used to reach the cluster from the notebook when available. This URL cannot be used to reach the cluster from a server.

PATCH/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}

curl -X PATCH \
  -H "X-Auth-Token: $SCW_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{}' \
  "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}"

Example Request Body

{
  "name": "name",
  "description": "description",
  "tags": [
    "string"
  ],
  "node_count": 0
}

json

Example Responses

{
  "id": "6170692e-7363-616c-6577-61792e636f6d",
  "project_id": "6170692e-7363-616c-6577-61792e636f6d",
  "name": "name",
  "description": "description",
  "tags": [
    "string"
  ],
  "main": {
    "node_type": "node_type",
    "spark_ui_url": "spark_ui_url",
    "spark_master_url": "spark_master_url",
    "root_volume": {
      "type": "unknown_type",
      "size": 0
    }
  },
  "worker": {
    "node_type": "node_type",
    "node_count": 0,
    "root_volume": {
      "type": "unknown_type",
      "size": 0
    }
  },
  "status": "unknown_status",
  "created_at": "2022-03-22T12:34:56.123456Z",
  "updated_at": "2022-03-22T12:34:56.123456Z",
  "region": "region",
  "has_notebook": true,
  "notebook_url": "notebook_url",
  "spark_version": "spark_version",
  "total_storage": {
    "type": "unknown_type",
    "size": 0
  },
  "private_network_id": "6170692e-7363-616c-6577-61792e636f6d",
  "notebook_master_url": "notebook_master_url"
}

json

application/json

GET

https://api.scaleway.com

/datalab/v1beta1/regions/{region}/node-types

List the available compute node types for creating a new cluster.

path Parameters

region

string · enum · required

The region you want to target

Enum values:

fr-par

it-mil

query Parameters

page

integer · int32

The page number.

page_size

integer · uint32

The page size.

order_by

scaleway.datalab.v1beta1.ListNodeTypesRequest.OrderBy · enum

The order by field. Available fields are name_asc, name_desc, vcpus_asc, vcpus_desc, memory_gigabytes_asc, memory_gigabytes_desc, vram_bytes_asc, vram_bytes_desc, gpus_asc, gpus_desc.

Enum values:

name_asc

name_desc

vcpus_asc

vcpus_desc

memory_gigabytes_asc

memory_gigabytes_desc

vram_bytes_asc

vram_bytes_desc

Default: name_asc

targets

scaleway.datalab.v1beta1.NodeType.Target[]

Filter based on the target of the nodes. Allows to filter the nodes based on their purpose which can be main or worker node.

Enum values:

unknown_target

notebook

worker

resource_type

scaleway.datalab.v1beta1.ListNodeTypesRequest.ResourceType · enum

Filter based on node type ( cpu/gpu/all ).

Enum values:

all

gpu

cpu

Default: all

Responses

200

scaleway.datalab.v1beta1.ListNodeTypesResponse

scaleway.datalab.v1beta1.NodeType[]

The list of node types.

total_count

integer · uint64

The total count of node types.

GET

https://api.scaleway.com

/datalab/v1beta1/regions/{region}/notebook-versions

Lists available notebook versions.

path Parameters

region

string · enum · required

The region you want to target

Enum values:

fr-par

it-mil

query Parameters

page

integer · int32

The page number.

page_size

integer · uint32

The page size.

order_by

scaleway.datalab.v1beta1.ListNotebookVersionsRequest.OrderBy · enum

The order by field. Available options are name_asc and name_desc.

Enum values:

name_asc

name_desc

Default: name_asc

Responses

200

scaleway.datalab.v1beta1.ListNotebookVersionsResponse

scaleway.datalab.v1beta1.Notebook[]

The list of notebook versions.

total_count

integer · uint64

The total count of notebook versions.