Clusters for Apache Kafka®

Introduction

Clusters for Apache Kafka® is Scaleway's product to provide a secure, fully-managed infrastructure for your Apache Kafka® deployments. With our clusters, you can create data pipelines and event-driven architectures that run natively on Apache Kafka®, without having to manage the underlying infrastructure.

The product is currently in Public Beta.

Concepts

Refer to our dedicated concepts page to find definitions of the different terms referring to Clusters for Apache Kafka®.

Quickstart

1. Configure your environment variables.

   Note

   This is an optional step that seeks to simplify your usage of the Kafka API.

   export SCW_SECRET_KEY="<API secret key>"
   export SCW_DEFAULT_ZONE="<Scaleway default Availability Zone>"
   export SCW_DEFAULT_REGION="<Scaleway default region>"
   export SCW_PROJECT_ID="<Scaleway Project ID>"
   

2. Create a cluster: Run the following command to create a cluster of 3 nodes with 20GB storage capacity per node. You can customize the details in the payload to your needs, using the table below to help. Note that you will need to have a VPC and a Private Network before running this command.

   curl -X POST \
     -H "X-Auth-Token: $SCW_SECRET_KEY" \
     -H "Content-Type: application/json" \
     -d '{
       "version": "4.0.0",
       "node_amount": 3,
       "name": "MyCluster",
       "user_name": "MyUser
       "password": "MySecurePassword",
       "project_id": $SCW_PROJECT_ID,
       "volume": {
           "size_bytes": 20000000000,
           "type": "sbs_5k"
       },
       "endpoints": [
           {
               "private_network": {
                   "private_network_id": {Your PN ID}
               }
           }
       ]
     }' \
     "https://api.scaleway.com/kafka/v1alpha1/regions/fr-par/clusters"
   

3. Get a list of your clusters: Run the following command to get a list of all the clusters in your account, with their details:

   curl -X GET \
     -H "Content-Type: application/json" \
     -H "X-Auth-Token: $SCW_SECRET_KEY" \
     "https://api.scaleway.com/kafka/v1alpha1/regions/fr-par/clusters"
   

4. Delete your cluster: Run the following command to delete a cluster. Ensure that you replace {resource-id} in the URL with the ID of the cluster you want to delete.

   curl -X DELETE \
     -H "Content-Type: application/json" \
     -H "X-Auth-Token: $SCW_SECRET_KEY" \
     "https://api.scaleway.com/kafka/v1alpha1/regions/fr-par/clusters/{resource-id}"
   

Technical information

Regions

Scaleway's infrastructure spans different regions and Availability Zones.

Clusters for Apache Kafka is currently available in the Paris region, which is represented by the following path parameter:

• fr-par

Technical limitations

Find out about product limitations in our FAQ.

Going further

For more information about Clusters for Apache Kafka®, you can check out the following pages:

• Clusters for Apache Kafka® Documentation
• Contact our support team.