Securing a cluster with a Private Network

Scaleway Kubernetes Kapsule provides a managed environment to create, configure and run a cluster of preconfigured machines for containerized applications. This allows you to create Kubernetes clusters without the complexity of managing the infrastructure.

All new Kubernetes clusters are deployed with a Scaleway Private Network using controlled isolation.

By default, worker nodes are currently delivered with public IP addresses (controlled isolation). These IPs serve solely for outgoing traffic from your nodes to the internet; no default services are set to listen on them. Even though these nodes have public IP addresses for specific maintenance and operational purposes, your cluster’s security remains uncompromised. See below for more information. Optionally, you can configure your nodes inside an entirely private network using full isolation.

Why have a Private Network for your Kubernetes Kapsule cluster?

A Private Network offers crucial functionalities to your cluster, including:

• Implementation of best practices in terms of security: all Scaleway resources can communicate securely (Instances, Load Balancers, Managed Databases), with less surface area for attack. For further information, refer to our blog post 10 best practices to configure your VPC.
• Compliance with expectations from the market (enterprise customers)
• Less manual configuration work such as security group configuration, IP range configuratio, etc.
• Multi-AZ compatibility allows you to create node pools in several Availability Zones for better resilience.
• Lower latency

How can I migrate my existing clusters to Private Networks?

1. In the Kubernetes section of the Scaleway console, locate your public clusters in the list.
2. Click the cluster’s name to navigate to your cluster information.
3. Use the recommended migration action in your cluster information, or go to the Private Network tab to start the migration.
   Note:

   Migrating an existing cluster is only available for clusters with a Kubernetes version above v1.22.0. Note that once a cluster is migrated, you cannot roll back this change.

What will happen after starting the migration:

1. Your control plane will restart for a first time: the Kubernetes API of your cluster will temporarily be unavailable.
2. Your pools will be upgraded to migrate the nodes into the Private Network. All of your nodes will be rebooted according to the specified Upgrade Policy of your pools.
3. Once all your nodes have rebooted, your control plane will be configured to use the Private Network and restarted: your Load Balancers will be reconfigured and migrated to the Private Network.
4. The CNI of your cluster will be reconfigured and restarted to use the Private Network.
   Important:

   During step IV, the Pod network of your cluster will be temporarily unavailable for 1 to 10 minutes as all Pods of the CNI are restarted, depending on the size of your cluster and the CNI you are using. Your Pods will be unable to communicate with each other during this step.

To further improve the security of your Kubernetes cluster, you can configure a security group to block inbound traffic on the public network interface of your nodes. However, be careful when updating the security group: stateful group must be enabled and if you currently use Kubernetes services with NodePorts, need SSH access, or need ICMP access, you must add a rule to allow this traffic.

What is the difference between controlled isolation and full isolation?

Worker node pools with controlled isolation inside a Private Network have both private and public IPs for outgoing traffic. Fully isolated nodes get only a private IP, with all external communications channeled through a Public Gateway for secure external connections.

What are the migration strategies provided?

As we transition towards Private Networks for Kapsule clusters, we understand that you may have varying needs and preferences when it comes to migrating your existing clusters.

We recommend two distinct migration strategies tailored to various requirements and limitations.

Scaleway product compatibility

Can I use a Public Gateway with my Private Network to exit all outgoing traffic from the nodes?

Yes, you are required to attach a Private Gateway when setting up a node pool with full isolation. This allows Kapsule nodes with private IPs to route their outgoing traffic through the Public Gateway. For detailed steps on setting up a Public Gateway, refer to our Public Gateway documentation. Keep in mind that removing or detaching the Public Gateway from the Private Network can cause a single point of failure in the cluster, preventing fully isolated node pools from accessing the control plane.

Is Kosmos compatible with Private Networks?

Only Kapsule can use a Private Network.

Kosmos uses Kilo as a CNI, which uses WireGuard to create a VPN Mesh between nodes for communication between pods. Any node in Kosmos, either in Scaleway or outside, uses these VPN tunnels to communicate securely by construct.

Are Managed Databases compatible with Kubernetes Kapsule on Private Networks?

Yes they are.

Since July 2023, the automatic allocation of IP addresses via IPAM is available for Managed Databases. These IP addresses are compatible with Scaleway’s VPC, which is now in General Availability. For more information about product compatibility refer to the VPC documentation.

For any new Private Networks you create and attach to Managed Databases after July 2023, your private IP addresses are automatically allocated.

If you have set up Private Network endpoints for your Managed Databases prior to July 2023, and want to connect to Kapsule via a Private Network, you must first delete your old private network endpoint. Then, you can create a new one, either via the Scaleway console or API.

In the example below we show you how to do so via the API. We specify the automated configuration of your Private Network via IPAM using "ipam_config": {},.

curl --request POST \
  --url https://api.scaleway.com/rdb/v1/regions/$REGION/instances/$INSTANCE_ID/endpoints \
  --header "Content-Type: application/json" \
  --header "X-Auth-Token: $SCW_SECRET_KEY" \
  --data '{
	"endpoint_spec": {
		"private_network": {
			"ipam_config": {},
			"private_network_id": "<PRIVATE_NETWORK_ID>"
		}
	}
}'Copy

Refer to the Managed Database for PostgreSQL and MySQL API documentation for further information.

Are managed Load Balancers compatible with Kubernetes Kapsule Private Networks?

Managed Load Balancers support Private Networks with private backends and public frontends, meaning the traffic is forwarded to your worker nodes through your clusters’ Private Network. Additionally, private Load Balancers are supported. These Load Balancers have no public IPs in either their back or frontends.

Which IP ranges are used for the Private Network of my cluster?

We automatically assign a /22 IP subnet from a Private Network to your cluster.

How can I access my cluster via my nodes’ public IPs for specific use cases?

Once you create a cluster in Kapsule, all nodes, particularly those with the Private Network feature enabled, are protected by a security group named kubernetes-<cluster-id>. Any changes made to this security group will be applicable to all nodes in the cluster.

If you wish to allow access to the nodes through a public IP using a specific port/protocol, you can modify the security group after creating the cluster by following these steps:

From the Scaleway console

1. Go to the Instances section of the Scaleway console.
2. Click the Security groups tab. A list of your existing security groups displays.
3. Click the name of the security group that is configured for your Instance, which is named kubernetes-<cluster-id>.
4. Click the Rules tab. A list of rules configured for this group displays.
5. Click «Edit Icon» to edit the security group rules.
6. Click Add inbound route to configure a new rule and customize it according to your requirements.
7. Apply your custom rules by clicking «Validate Icon».

Using Terraform

If you are using Terraform to create your cluster, you can create a security group resource after creating the cluster resource and before creating the pool resource. You can find a Terraform configuration example below:

data "scaleway_k8s_version" "latest" {
name = "latest"
}
resource "scaleway_vpc_private_network" "kapsule" {
  name = "pn_kapsule"
  tags = ["kapsule"]
}
resource "scaleway_k8s_cluster" "kapsule" {
  name                        = "open-pn-test"
  version                     = data.scaleway_k8s_version.latest.name
  cni                         = "cilium"
  private_network_id          = scaleway_vpc_private_network.kapsule.id
  delete_additional_resources = true
  depends_on                  = [scaleway_vpc_private_network.kapsule]
}
resource "scaleway_instance_security_group" "kapsule" {
  name                    = "kubernetes ${split("/", scaleway_k8s_cluster.kapsule.id)[1]}"
  inbound_default_policy  = "drop"
  outbound_default_policy = "accept"
  stateful                = true
  inbound_rule {
    action   = "accept"
    protocol = "UDP"
    port     = "500"
  }
  depends_on              = [scaleway_k8s_cluster.kapsule]
}
resource "scaleway_k8s_pool" "default" {
  cluster_id          = scaleway_k8s_cluster.kapsule.id
  name                = "default"
  node_type           = "DEV1-M"
  size                = 1
  autohealing         = true
  wait_for_pool_ready = true
  depends_on          = [scaleway_instance_security_group.kapsule]
}
resource "scaleway_rdb_instance" "main" {
  name           = "pn-rdb"
  node_type      = "DB-DEV-S"
  engine         = "PostgreSQL-14"
  is_ha_cluster  = true
  disable_backup = true
  user_name      = "username"
  password       = "thiZ_is_v&ry_s3cret" # Obviously change password here or generate one at runtime through null_resource and display it via output.
  private_network {
    pn_id = scaleway_vpc_private_network.kapsule.id
  }
}Copy

Will the control plane also be located inside the Private Network?

Currently, only worker nodes are located in the Private Network of your cluster. The communication between the nodes and the control plane uses the Public IP of the node. You can reach the control plane for nodes using full isolation by adding a Public Gateway to the cluster.

What future options will there be for isolation?

• Control plane in isolation with nodes and API communicating in the same isolated network. The CNI’s network policies will restrict/allow a range of IPs or ports to control who can access the API server.
• End of support for legacy public clusters. All clusters will need to migrate to controlled isolation (Private Networks) by Q1/2024.