Clusters for Apache Kafka® FAQ
General
What is Clusters for Apache Kafka®?
Clusters for Apache Kafka® is Scaleway's product to provide a secure, fully-managed infrastructure for your Apache Kafka® deployments. With our clusters, you can create data pipelines and event-driven architectures that run natively on Apache Kafka®, without having to manage the underlying infrastructure. This product is currently in Private Beta.
How do I use my cluster, and how can I connect to it?
To use your cluster, you first connect via the Kafka CLI or any Kafka-supported programming language. The connection is secured, and requires the appropriate credentials, certificate, and endpoint provided by Scaleway. All configuration is done through a Kafka client, allowing you to manage topics, producers, and consumers easily. This flexibility ensures seamless integration with your applications and workflows.
What limitations apply during the Private Beta?
Private Beta is aimed at providing an early version of a future product, to allow users to test and validate it during its development. The following limitations apply to Clusters for Apache Kafka® in its current Private Beta phase:
- Only selected users can test the product (request an invitation).
- Only one Kafka version is supported.
- Clusters can only be provisioned with either one or three nodes (brokers).
- The number of nodes cannot be modified after creation of the cluster.
- Storage is fixed at 100 GB per node.
- Connectivity to the cluster is facilitated exclusively through Private Networks.
- Only one set of credentials (username and password) can be created per cluster.
- Each Scaleway Organization in the Private Beta may have a maximum of 5 clusters at one time.
We aim to improve and develop the product based on feedback received during the beta stage.
Is Scaleway's Clusters for Apache Kafka® based on the open-source Kafka framework?
Yes, Scaleway's Clusters for Apache Kafka is built on the open-source Apache Kafka framework. We use the Kafka API to ensure compatibility and leverage the robust features of the open-source ecosystem.
How is Apache Kafka® different to RabbitMQ and NATS?
Apache Kafka, RabbitMQ, and NATS are all messaging systems but serve different purposes and have distinct architectures.
- Apache Kafka is designed for high-throughput, real-time data streaming, and is topic-based, using the pull model. It can process large amounts of data in real-time with high concurrency and fault tolerance, and is ideal for stream processing, event sourcing, and big data applications.
- RabbitMQ is a versatile message broker that supports multiple protocols, and offers strong support for message delivery guarantees and reliability. It uses the push model, and is well-suited for traditional message queuing and task distribution.
- NATS is a lightweight, high-performance subject-based messaging system focused on speed and simplicity, making it great for microservices.
Each system has its own strengths and is suited to different use cases depending on the requirements of the application.
Billing
How am I billed for Clusters for Apache Kafka®?
Clusters for Apache Kafka® are free of charge during Private Beta, and you will not be billed. After the Private Beta, we are considering a usage-based pricing model that will be competitive with current market offerings.