Scaleway logo
Clusters for Apache Kafka®

Clusters for Apache Kafka®

Download schema

Clusters for Apache Kafka® is Scaleway's product to provide a secure, fully-managed infrastructure for your Apache Kafka® deployments. With our clusters, you can create data pipelines and event-driven architectures that run natively on Apache Kafka®, without having to manage the underlying infrastructure.

Concepts

Refer to our dedicated concepts pageOpen in new context to find definitions of the different terms referring to Clusters for Apache Kafka®.

Quickstart

  1. Configure your environment variables.

    Note

    This is an optional step that seeks to simplify your usage of the Kafka API.

    TerminalCode
    export SCW_SECRET_KEY="<API secret key>"
export SCW_DEFAULT_ZONE="<Scaleway default Availability Zone>"
export SCW_DEFAULT_REGION="<Scaleway default region>"
export SCW_PROJECT_ID="<Scaleway Project ID>"

  2. Create a cluster: Run the following command to create a cluster with a node on each AZ (3 nodes by default) with 20GB storage capacity per node. You can customize the details in the payload to your needs, using the table below to help. Note that you will need to have a VPC and a Private NetworkOpen in new context before running this command.

    TerminalCode
    curl -X POST \
  -H "X-Auth-Token: $SCW_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "version": "4.0.0",
    "node_amount": 3,
    "multi_az": {},
    "name": "MyCluster",
    "user_name": "MyUser
    "password": "MySecurePassword",
    "project_id": $SCW_PROJECT_ID,
    "volume": {
        "size_bytes": 20000000000,
        "type": "sbs_5k"
    },
    "endpoints": [
        {
            "private_network": {
                "private_network_id": {Your PN ID}
            }
        }
    ]
  }' \
  "https://api.scaleway.com/kafka/v1alpha1/regions/fr-par/clusters"

  3. Get a list of your clusters: Run the following command to get a list of all the clusters in your account, with their details:

    TerminalCode
    curl -X GET \
  -H "Content-Type: application/json" \
  -H "X-Auth-Token: $SCW_SECRET_KEY" \
  "https://api.scaleway.com/kafka/v1alpha1/regions/fr-par/clusters"

  4. Delete your cluster: Run the following command to delete a cluster. Ensure that you replace {resource-id} in the URL with the ID of the cluster you want to delete.

    TerminalCode
    curl -X DELETE \
  -H "Content-Type: application/json" \
  -H "X-Auth-Token: $SCW_SECRET_KEY" \
  "https://api.scaleway.com/kafka/v1alpha1/regions/fr-par/clusters/{resource-id}"

Technical information

Regions

Scaleway's infrastructure spans different regions and Availability ZonesOpen in new context.

Clusters for Apache Kafka is currently available in the Paris region, which is represented by the following path parameter:

  • fr-par

Technical limitations

Find out about product limitations in our FAQOpen in new context.

Going further

For more information about Clusters for Apache Kafka®, you can check out the following pages: