Clusters for Apache Kafka® - Concepts
Cluster
A cluster is a deployment of Apache Kafka®. It consists of one or more nodes (called brokers) that work together to store, manage and distribute event streams. Data replication across the cluster ensures high availability and robustness.
Clusters for Apache Kafka®
Clusters for Apache Kafka® is Scaleway's product to provide a secure, fully-managed infrastructure for your Apache Kafka® deployments. With our clusters, you can create data pipelines and event-driven architectures that run natively on Apache Kafka®, without having to manage the underlying infrastructure.
Credentials
Credentials consist of a username and password that allow you to connect to your cluster. These are defined during cluster creation. During Private Beta, a limit of one user (i.e. one set of credentials) per cluster is applied.
Data replication
Data replication is the mechanism that ensures event data is copied across multiple brokers in a cluster, for fault tolerance and high availability. If a node fails, data replication means Apache Kafka® can keep serving data without downtime or data loss.
Endpoint
During Private Beta, access to your Cluster for Apache Kafka® is via private endpoints only. When creating your cluster, you must attach it to a Private Network, and all nodes in the cluster receive a private IPv4 endpoint on this network. Endpoints are in a format similar to bf610fb0-f7eb-4ec3-a893...634aea.kafka.fr-par.scw.cloud.
Event streaming
Clusters for Apache Kafka® is specifically designed for implementing event streaming at scale. Event streaming is the process of capturing, storing and processing a continuous flow of events in real time. Each event represents something that happened, e.g.User added item to cart, User clicked 'Pay now', Payment was rejected by bank. Events are stored in topics, e.g. Inventory, Payment, Orders. Typically, programs and microservices subscribe to these topics to consume their events and take action in relation to them, e.g. Generate invoice, Check for fraud, Program delivery.
Node
A Cluster for Apache Kafka® consists of one or more nodes, also called brokers. Nodes work together to store, manage, and distribute event streams. When creating a cluster, you can choose between dedicated nodes or shared nodes, based on your workload requirements, budget, and use case:
- Dedicated node: Runs only your cluster, with no other customers sharing the broker's resources.
- Shared node: Runs your cluster along with other customers', on the same broker.
Storage
A cluster's storage is used to persist events, so they can be reliably consumed by services and applications. During Private Beta, each Cluster for Apache Kafka® has a fixed amount of 100 GB of storage per node.
TLS certificate
A TLS certificate (aka SSL certificate) is a digital credential that encrypts data in transit and verifies the identity of a server or client. Scaleway generates a TLS certificate for each Cluster for Apache Kafka®, and all clients connecting to the cluster must install this certificate in order to be authenticated. This ensures secure and encrypted communication between nodes (brokers) and clients.
Users
See credentials.