openapi: 3.1.0 info: title: Clusters for Apache Spark™ API description: "Clusters for Apache Spark™ is Scaleway's fully-managed service for running Apache Spark™ workloads. It provides a scalable, secure environment to process large datasets with ease. With Clusters for Apache Spark™, you can launch Apache Spark™ clusters in minutes and focus on data processing instead of infrastructure management.\n\nThe service is currently in General Availability. \n\n## Concepts\n\nRefer to our [dedicated concepts page](https://www.scaleway.com/en/docs/data-lab/concepts/) to find definitions of the different terms referring to Clusters for Apache Spark™.\n\n## Quickstart\n\n1. Configure your environment variables.\n\n \n \ This is an optional step that seeks to simplify your usage of the Clusters for Apache Spark™ API.\n \n\n ```bash\n export SCW_SECRET_KEY=\"\"\n export SCW_DEFAULT_ZONE=\"\"\n \ export SCW_DEFAULT_REGION=\"\"\n export SCW_PROJECT_ID=\"\"\n ```\n\n2. **Create a new cluster**: Run the following command to create a cluster with a main node and 2 worker nodes with 20GB of total persistent volume storage. You can customize the details in the payload to your needs, using the table below to help. Note that you will need to have a [VPC and a Private Network](https://www.scaleway.com/en/developers/api/vpc/) before running this command.\n\n ```bash\n curl --request POST \\\n --url https://api.scaleway.com//datalab/v1beta1/regions/fr-par/datalabs \\\n -H \"X-Auth-Token: $SCW_SECRET_KEY\" \\\n -H \"Content-Type: application/json\" -d '{\n \"name\": \"my-first-cluster\",\n \"project_id\": \"'\"$SCW_PROJECT_ID\"'\",\n \"worker\": {\n \"node_type\": \"DDL-POP2-2C-8G\",\n \"node_count\": 2\n },\n \ \"main\": {\n \"node_type\": \"DDL-PLAY2-MICRO\"\n \ }, \"has_notebook\": true,\n \"total_storage\": {\n \ \"size\": 20000000000,\n \"type\": \"sbs_5k\"\n },\n \"private_network_id\": \"{Your PN ID}\",\n \"spark_version\": \"4.0.0\"\n }\n }'\n ```\n\n3. **Get a list of your clusters**: Run the following command to get a list of all the clusters in your account, with their details:\n\n ```bash\n curl --request GET \\\n --url https://api.scaleway.com/datalab/v1beta1/regions/fr-par/datalabs \\\n -H \"X-Auth-Token: $SCW_SECRET_KEY\"\n ```\n\n4. **Delete your cluster**: Run the following command to delete a cluster. Ensure that you replace `{datalab-id}` in the URL with the ID of the cluster you want to delete.\n\n ```bash\n curl --request DELETE \\\n --url https://api.scaleway.com/datalab/v1beta1/regions/fr-par/datalabs/{datalab-id} \\\n -H \"X-Auth-Token: $SCW_SECRET_KEY\" | jq\n ```\n\n## Technical information\n\n### Regions\n\nScaleway's infrastructure spans different [regions and Availability Zones](https://www.scaleway.com/en/docs/account/reference-content/products-availability/).\n\nClusters for Apache Spark™ is currently available in the Paris & Milano regions, which is represented by the following path parameter:\n\n- fr-par\n- it-mil\n\n## Going further\n\nFor more information about Clusters for Apache Spark™, you can check out the following pages:\n\n* [Clusters for Apache Spark™ Documentation](https://www.scaleway.com/en/docs/data-lab/)\n* [Contact our support team](https://console.scaleway.com/support/tickets)." version: v1beta1 servers: - url: https://api.scaleway.com tags: - name: Clusters for Apache Spark™ description: | Each cluster is composed of one or more dedicated compute nodes running Apache Spark™. It can optionally include a dedicated JupyterLab Notebook. These resources are fully manageable through this API. components: schemas: scaleway.datalab.v1beta1.Cluster: type: object properties: name: type: string description: The name of the cluster. description: type: string description: The description of the cluster. versions: type: array description: The versions of the cluster. items: $ref: '#/components/schemas/scaleway.datalab.v1beta1.ClusterVersion' x-properties-order: - name - description - versions scaleway.datalab.v1beta1.ClusterVersion: type: object properties: version: type: string description: The version of the cluster. end_of_life: type: string description: The end of life date of the cluster version. (RFC 3339 format) format: date-time example: "2022-03-22T12:34:56.123456Z" nullable: true created_at: type: string description: The creation date of the cluster version. (RFC 3339 format) format: date-time example: "2022-03-22T12:34:56.123456Z" nullable: true updated_at: type: string description: The last update date of the cluster version. (RFC 3339 format) format: date-time example: "2022-03-22T12:34:56.123456Z" nullable: true disabled: type: boolean description: Whether the cluster version is disabled. beta: type: boolean description: Whether the cluster version is in beta. x-properties-order: - version - end_of_life - created_at - updated_at - disabled - beta scaleway.datalab.v1beta1.Datalab: type: object properties: id: type: string description: The unique identifier of the cluster. (UUID format) example: 6170692e-7363-616c-6577-61792e636f6d project_id: type: string description: The unique identifier of the project where the cluster has been created. (UUID format) example: 6170692e-7363-616c-6577-61792e636f6d name: type: string description: The name of the cluster. description: type: string description: The description of the cluster. tags: type: array description: The tags of the cluster. items: type: string main: type: object description: The Apache Spark™ Main node specification of cluster. It holds the parameters `node_type`, `spark_ui_url` (available to reach Apache Spark™ UI), `spark_master_url` (used to reach the cluster within a VPC), `root_volume` (size of the volume assigned to the cluster). properties: node_type: type: string spark_ui_url: type: string spark_master_url: type: string root_volume: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Volume' x-properties-order: - node_type - spark_ui_url - spark_master_url - root_volume worker: type: object description: The cluster worker nodes specification. It holds the parameters `node_type`, `node_count`, `root_volume` (size of the volume assigned to the cluster). properties: node_type: type: string node_count: type: integer format: uint32 root_volume: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Volume' x-properties-order: - node_type - node_count - root_volume status: type: string description: The status of the cluster. For a working cluster the status is marked as `ready`. enum: - unknown_status - creating - updating - ready - error - deleting - locked - deleted default: unknown_status created_at: type: string description: The creation timestamp of the cluster. (RFC 3339 format) format: date-time example: "2022-03-22T12:34:56.123456Z" nullable: true updated_at: type: string description: The last update date of the cluster. (RFC 3339 format) format: date-time example: "2022-03-22T12:34:56.123456Z" nullable: true region: type: string description: The region of the cluster. has_notebook: type: boolean description: Whether a JupyterLab notebook is associated with the cluster or not. notebook_url: type: string description: The URL of the notebook if available. nullable: true spark_version: type: string description: The version of Apache Spark™ running inside the cluster. total_storage: type: object description: The total persistent volume storage selected to run Apache Spark™. properties: type: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Volume.Type' size: type: integer description: (in bytes) format: uint64 x-properties-order: - type - size private_network_id: type: string description: The unique identifier of the private network to which the cluster is attached to. (UUID format) example: 6170692e-7363-616c-6577-61792e636f6d notebook_master_url: type: string description: The URL that is used to reach the cluster from the notebook when available. This URL cannot be used to reach the cluster from a server. nullable: true x-properties-order: - id - project_id - name - description - tags - main - worker - status - created_at - updated_at - region - has_notebook - notebook_url - spark_version - total_storage - private_network_id - notebook_master_url scaleway.datalab.v1beta1.ListClusterVersionsResponse: type: object properties: clusters: type: array description: The list of cluster versions. items: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Cluster' total_count: type: integer description: The total count of cluster versions. format: uint64 x-properties-order: - clusters - total_count scaleway.datalab.v1beta1.ListDatalabsResponse: type: object properties: datalabs: type: array description: The list of clusters. This is a list composed of messages of type `DataLab`. items: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Datalab' total_count: type: integer description: The total count of clusters. format: uint64 x-properties-order: - datalabs - total_count scaleway.datalab.v1beta1.ListNodeTypesResponse: type: object properties: node_types: type: array description: The list of node types. items: $ref: '#/components/schemas/scaleway.datalab.v1beta1.NodeType' total_count: type: integer description: The total count of node types. format: uint64 x-properties-order: - node_types - total_count scaleway.datalab.v1beta1.ListNotebookVersionsResponse: type: object properties: notebooks: type: array description: The list of notebook versions. items: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Notebook' total_count: type: integer description: The total count of notebook versions. format: uint64 x-properties-order: - notebooks - total_count scaleway.datalab.v1beta1.NodeType: type: object properties: stock_status: type: string description: The stock status of the node type. enum: - unknown_stock - low_stock - out_of_stock - available default: unknown_stock name: type: string description: The name of the node type. description: type: string description: The description of the node type. vcpus: type: integer description: The number of vCPUs. format: uint32 memory_gigabytes: type: integer description: The amount of memory. format: uint64 vram_gigabytes: type: integer description: The amount of VRAM. format: uint64 gpus: type: integer description: The number of GPUs. format: uint32 disabled: type: boolean description: Whether the node type is disabled. beta: type: boolean description: Whether the node type is in beta. created_at: type: string description: The creation date of the node type. (RFC 3339 format) format: date-time example: "2022-03-22T12:34:56.123456Z" nullable: true updated_at: type: string description: The last update date of the node type. (RFC 3339 format) format: date-time example: "2022-03-22T12:34:56.123456Z" nullable: true targets: type: array description: The targets of the node type. items: $ref: '#/components/schemas/scaleway.datalab.v1beta1.NodeType.Target' x-properties-order: - stock_status - name - description - vcpus - memory_gigabytes - vram_gigabytes - gpus - disabled - beta - created_at - updated_at - targets scaleway.datalab.v1beta1.NodeType.Target: type: string enum: - unknown_target - notebook - worker default: unknown_target scaleway.datalab.v1beta1.Notebook: type: object properties: name: type: string description: The name of the notebook. description: type: string description: The description of the notebook. versions: type: array description: The versions of the notebook. items: $ref: '#/components/schemas/scaleway.datalab.v1beta1.NotebookVersion' x-properties-order: - name - description - versions scaleway.datalab.v1beta1.NotebookVersion: type: object properties: version: type: string description: The version of the notebook. end_of_life: type: string description: The end of life date of the notebook version. (RFC 3339 format) format: date-time example: "2022-03-22T12:34:56.123456Z" nullable: true created_at: type: string description: The creation date of the notebook version. (RFC 3339 format) format: date-time example: "2022-03-22T12:34:56.123456Z" nullable: true updated_at: type: string description: The last update date of the notebook version. (RFC 3339 format) format: date-time example: "2022-03-22T12:34:56.123456Z" nullable: true disabled: type: boolean description: Whether the notebook version is disabled. beta: type: boolean description: Whether the notebook version is in beta. x-properties-order: - version - end_of_life - created_at - updated_at - disabled - beta scaleway.datalab.v1beta1.Volume: type: object properties: type: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Volume.Type' size: type: integer description: (in bytes) format: uint64 x-properties-order: - type - size scaleway.datalab.v1beta1.Volume.Type: type: string enum: - unknown_type - sbs_5k default: unknown_type securitySchemes: scaleway: in: header name: X-Auth-Token type: apiKey paths: /datalab/v1beta1/regions/{region}/cluster-versions: get: tags: - Clusters for Apache Spark™ operationId: ListClusterVersions description: List the Apache Spark™ versions the product is compatible with. parameters: - in: path name: region description: The region you want to target required: true schema: type: string enum: - fr-par - it-mil - in: query name: page description: The page number. schema: type: integer format: int32 - in: query name: page_size description: The page size. schema: type: integer format: uint32 - in: query name: order_by description: The order by field. schema: type: string enum: - name_asc - name_desc default: name_asc responses: "200": description: "" content: application/json: schema: $ref: '#/components/schemas/scaleway.datalab.v1beta1.ListClusterVersionsResponse' security: - scaleway: [] x-codeSamples: - lang: cURL source: |- curl -X GET \ -H "X-Auth-Token: $SCW_SECRET_KEY" \ "https://api.scaleway.com/datalab/v1beta1/regions/{region}/cluster-versions" - lang: HTTPie source: |- http GET "https://api.scaleway.com/datalab/v1beta1/regions/{region}/cluster-versions" \ X-Auth-Token:$SCW_SECRET_KEY /datalab/v1beta1/regions/{region}/datalabs: get: tags: - Clusters for Apache Spark™ operationId: ListDatalabs description: List information about cluster within a project or an organization. parameters: - in: path name: region description: The region you want to target required: true schema: type: string enum: - fr-par - it-mil - in: query name: organization_id description: The unique identifier of the organization whose clusters you want to list. schema: type: string - in: query name: project_id description: The unique identifier of the project whose clusters you want to list. schema: type: string - in: query name: name description: The name of the cluster you want to list. schema: type: string - in: query name: tags description: The tags associated with the cluster you want to list. schema: type: array items: type: string - in: query name: page description: The page number for pagination. schema: type: integer format: int32 - in: query name: page_size description: The page size for pagination. schema: type: integer format: uint32 - in: query name: order_by description: The order by field, available options are `name_asc`, `name_desc`, `created_at_asc`, `created_at_desc`, `updated_at_asc`, `updated_at_desc`. schema: type: string enum: - name_asc - name_desc - created_at_asc - created_at_desc - updated_at_asc - updated_at_desc default: name_asc responses: "200": description: "" content: application/json: schema: $ref: '#/components/schemas/scaleway.datalab.v1beta1.ListDatalabsResponse' security: - scaleway: [] x-codeSamples: - lang: cURL source: |- curl -X GET \ -H "X-Auth-Token: $SCW_SECRET_KEY" \ "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs" - lang: HTTPie source: |- http GET "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs" \ X-Auth-Token:$SCW_SECRET_KEY post: tags: - Clusters for Apache Spark™ operationId: CreateDatalab description: Create a new cluster. In this call, one can personalize the node counts, add a notebook, choose the private network, define the persistent volume storage capacity. parameters: - in: path name: region description: The region you want to target required: true schema: type: string enum: - fr-par - it-mil responses: "200": description: "" content: application/json: schema: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Datalab' requestBody: required: true content: application/json: schema: type: object properties: project_id: type: string description: The unique identifier of the project where the cluster will be created. name: type: string description: The name of the cluster. description: type: string description: The description of the cluster. tags: type: array description: The tags of the cluster. items: type: string main: type: object description: The cluster main node specification. It holds the parameters `node_type` which specifies the node type of the main node. See ListNodeTypes for available options. See ListNodeTypes for available options. properties: node_type: type: string x-properties-order: - node_type worker: type: object description: The cluster worker node specification. It holds the parameters `node_type` which specifies the node type of the worker node and `node_count` for specifying the amount of nodes. properties: node_type: type: string node_count: type: integer format: uint32 x-properties-order: - node_type - node_count has_notebook: type: boolean description: Select this option to include a notebook as part of the cluster. spark_version: type: string description: The version of Apache Spark™ running inside the cluster, available options can be viewed at ListClusterVersions. total_storage: type: object description: The maximum persistent volume storage that will be available during workload. properties: type: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Volume.Type' size: type: integer description: (in bytes) format: uint64 x-properties-order: - type - size private_network_id: type: string description: The unique identifier of the private network the cluster will be attached to. x-properties-order: - project_id - name - description - tags - main - worker - has_notebook - spark_version - total_storage - private_network_id security: - scaleway: [] x-codeSamples: - lang: cURL source: |- curl -X POST \ -H "X-Auth-Token: $SCW_SECRET_KEY" \ -H "Content-Type: application/json" \ -d '{ "description": "string", "has_notebook": false, "name": "string", "private_network_id": "string", "project_id": "string", "spark_version": "string" }' \ "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs" - lang: HTTPie source: |- http POST "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs" \ X-Auth-Token:$SCW_SECRET_KEY \ description="string" \ has_notebook:=false \ name="string" \ private_network_id="string" \ project_id="string" \ spark_version="string" /datalab/v1beta1/regions/{region}/datalabs/{datalab_id}: get: tags: - Clusters for Apache Spark™ operationId: GetDatalab description: Retrieve information about a given cluster, specified by the `region` and `datalab_id` parameters. Its full details, including name, status, node counts, are returned in the response object. parameters: - in: path name: region description: The region you want to target required: true schema: type: string enum: - fr-par - it-mil - in: path name: datalab_id description: The unique identifier of the cluster. required: true schema: type: string responses: "200": description: "" content: application/json: schema: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Datalab' security: - scaleway: [] x-codeSamples: - lang: cURL source: |- curl -X GET \ -H "X-Auth-Token: $SCW_SECRET_KEY" \ "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}" - lang: HTTPie source: |- http GET "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}" \ X-Auth-Token:$SCW_SECRET_KEY patch: tags: - Clusters for Apache Spark™ operationId: UpdateDatalab description: Update a cluster node counts. Allows for up- and downscaling on demand, depending on the expected workload. parameters: - in: path name: region description: The region you want to target required: true schema: type: string enum: - fr-par - it-mil - in: path name: datalab_id description: The unique identifier of the cluster. required: true schema: type: string responses: "200": description: "" content: application/json: schema: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Datalab' requestBody: required: true content: application/json: schema: type: object properties: name: type: string description: The updated name of the cluster. nullable: true description: type: string description: The updated description of the cluster. nullable: true tags: type: array description: The updated tags of the cluster. items: type: string node_count: type: integer description: The updated node count of the cluster. Scale up or down the number of worker nodes. format: uint32 nullable: true x-properties-order: - name - description - tags - node_count security: - scaleway: [] x-codeSamples: - lang: cURL source: |- curl -X PATCH \ -H "X-Auth-Token: $SCW_SECRET_KEY" \ -H "Content-Type: application/json" \ -d '{}' \ "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}" - lang: HTTPie source: |- http PATCH "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}" \ X-Auth-Token:$SCW_SECRET_KEY delete: tags: - Clusters for Apache Spark™ operationId: DeleteDatalab description: Delete a cluster based on its region and id. parameters: - in: path name: region description: The region you want to target required: true schema: type: string enum: - fr-par - it-mil - in: path name: datalab_id description: The unique identifier of the cluster. required: true schema: type: string responses: "200": description: "" content: application/json: schema: $ref: '#/components/schemas/scaleway.datalab.v1beta1.Datalab' security: - scaleway: [] x-codeSamples: - lang: cURL source: |- curl -X DELETE \ -H "X-Auth-Token: $SCW_SECRET_KEY" \ "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}" - lang: HTTPie source: |- http DELETE "https://api.scaleway.com/datalab/v1beta1/regions/{region}/datalabs/{datalab_id}" \ X-Auth-Token:$SCW_SECRET_KEY /datalab/v1beta1/regions/{region}/node-types: get: tags: - Clusters for Apache Spark™ operationId: ListNodeTypes description: List the available compute node types for creating a new cluster. parameters: - in: path name: region description: The region you want to target required: true schema: type: string enum: - fr-par - it-mil - in: query name: page description: The page number. schema: type: integer format: int32 - in: query name: page_size description: The page size. schema: type: integer format: uint32 - in: query name: order_by description: The order by field. Available fields are `name_asc`, `name_desc`, `vcpus_asc`, `vcpus_desc`, `memory_gigabytes_asc`, `memory_gigabytes_desc`, `vram_bytes_asc`, `vram_bytes_desc`, `gpus_asc`, `gpus_desc`. schema: type: string enum: - name_asc - name_desc - vcpus_asc - vcpus_desc - memory_gigabytes_asc - memory_gigabytes_desc - vram_bytes_asc - vram_bytes_desc - gpus_asc - gpus_desc default: name_asc - in: query name: targets description: Filter based on the target of the nodes. Allows to filter the nodes based on their purpose which can be main or worker node. schema: type: array items: $ref: '#/components/schemas/scaleway.datalab.v1beta1.NodeType.Target' - in: query name: resource_type description: Filter based on node type ( `cpu`/`gpu`/`all` ). schema: type: string enum: - all - gpu - cpu default: all responses: "200": description: "" content: application/json: schema: $ref: '#/components/schemas/scaleway.datalab.v1beta1.ListNodeTypesResponse' security: - scaleway: [] x-codeSamples: - lang: cURL source: |- curl -X GET \ -H "X-Auth-Token: $SCW_SECRET_KEY" \ "https://api.scaleway.com/datalab/v1beta1/regions/{region}/node-types" - lang: HTTPie source: |- http GET "https://api.scaleway.com/datalab/v1beta1/regions/{region}/node-types" \ X-Auth-Token:$SCW_SECRET_KEY /datalab/v1beta1/regions/{region}/notebook-versions: get: tags: - Clusters for Apache Spark™ operationId: ListNotebookVersions description: Lists available notebook versions. parameters: - in: path name: region description: The region you want to target required: true schema: type: string enum: - fr-par - it-mil - in: query name: page description: The page number. schema: type: integer format: int32 - in: query name: page_size description: The page size. schema: type: integer format: uint32 - in: query name: order_by description: The order by field. Available options are `name_asc` and `name_desc`. schema: type: string enum: - name_asc - name_desc default: name_asc responses: "200": description: "" content: application/json: schema: $ref: '#/components/schemas/scaleway.datalab.v1beta1.ListNotebookVersionsResponse' security: - scaleway: [] x-codeSamples: - lang: cURL source: |- curl -X GET \ -H "X-Auth-Token: $SCW_SECRET_KEY" \ "https://api.scaleway.com/datalab/v1beta1/regions/{region}/notebook-versions" - lang: HTTPie source: |- http GET "https://api.scaleway.com/datalab/v1beta1/regions/{region}/notebook-versions" \ X-Auth-Token:$SCW_SECRET_KEY