Clusters for Apache Kafka®

Download schema

Clusters for Apache Kafka® is Scaleway's product to provide a secure, fully-managed infrastructure for your Apache Kafka® deployments. With our clusters, you can create data pipelines and event-driven architectures that run natively on Apache Kafka®, without having to manage the underlying infrastructure.

The product is currently in Public Beta.

Concepts

Refer to our dedicated concepts pageOpen in new context to find definitions of the different terms referring to Clusters for Apache Kafka®.

Quickstart

Configure your environment variables.

Note

This is an optional step that seeks to simplify your usage of the Kafka API.

Code
export SCW_SECRET_KEY="<API secret key>"
export SCW_DEFAULT_ZONE="<Scaleway default Availability Zone>"
export SCW_DEFAULT_REGION="<Scaleway default region>"
export SCW_PROJECT_ID="<Scaleway Project ID>"

Create a cluster: Run the following command to create a cluster with a node on each AZ (3 nodes by default) with 20GB storage capacity per node. You can customize the details in the payload to your needs, using the table below to help. Note that you will need to have a VPC and a Private NetworkOpen in new context before running this command.

Code
curl -X POST \
  -H "X-Auth-Token: $SCW_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "version": "4.0.0",
    "node_amount": 3,
    "multi_az": {},
    "name": "MyCluster",
    "user_name": "MyUser
    "password": "MySecurePassword",
    "project_id": $SCW_PROJECT_ID,
    "volume": {
        "size_bytes": 20000000000,
        "type": "sbs_5k"
    },
    "endpoints": [
        {
            "private_network": {
                "private_network_id": {Your PN ID}
            }
        }
    ]
  }' \
  "https://api.scaleway.com/kafka/v1alpha1/regions/fr-par/clusters"

Get a list of your clusters: Run the following command to get a list of all the clusters in your account, with their details:

Code
curl -X GET \
  -H "Content-Type: application/json" \
  -H "X-Auth-Token: $SCW_SECRET_KEY" \
  "https://api.scaleway.com/kafka/v1alpha1/regions/fr-par/clusters"

Delete your cluster: Run the following command to delete a cluster. Ensure that you replace {resource-id} in the URL with the ID of the cluster you want to delete.

Code
curl -X DELETE \
  -H "Content-Type: application/json" \
  -H "X-Auth-Token: $SCW_SECRET_KEY" \
  "https://api.scaleway.com/kafka/v1alpha1/regions/fr-par/clusters/{resource-id}"

Technical information

Regions

Scaleway's infrastructure spans different regions and Availability ZonesOpen in new context.

Clusters for Apache Kafka is currently available in the Paris region, which is represented by the following path parameter:

fr-par

Technical limitations

Find out about product limitations in our FAQOpen in new context.

Going further

For more information about Clusters for Apache Kafka®, you can check out the following pages: