Scaleway Private LB for Golem.ai Security Assets

At Golem.ai, our day-to-day decisions and choices justify the trust our customers place in us.

Security is at the heart of each and every one of them.

That's why we distinguish between public and private traffic at the very heart of our applications and our Kubernetes Hosting Platform operated by Scaleway, our French Cloud provider.

Thus, Public traffic, which by definition enables our customers to consume our applications, is strictly controlled by our powerful Web Application Firewall (WAF) and AntiDDoS, while private traffic is dedicated to the administration of our applications by our Teams, and is only accessible via a ZeroTrust VPN Tunnel connected to the Private LoadBalancer.

This gives us a 360° view of network activity upstream and downstream of the platform.

In terms of watertightness between our different STAGING, PREPRODUCTION, PRODUCTION and MONITORING environments, each has its own virtual private network (VLAN Layer 2 within a VPC).

This prevents private flows from being opened up to the outside world, and isolates flows between environments if necessary.

Endpoint : Public / Private Flow

Operation

The operations described below correspond to the needs of Golem.ai.
Adapt to your needs !

To use a Private LoadBalancer between our application and the Golem.ai Private Network, we'll need to perform the following steps in order.

1 . Create a new Private LB with WebUI , CLI or Terraform

Click Load Balancers in the Network section of the Scaleway console side menu. If you have not already created a Load Balancer, the product creation page is displayed. Otherwise, your list of existing Load Balancers displays. Then, choose "Private Load Balancer". More info here, and below...

Informations1. LB Name Fill in the name of the loadbalancer, respecting the nomenclature. private-lb-private-“env”-“domain without extension” eg . private-lb-prod-test for test.golem.ai2. LB Zone Select zone where application is localised (PARIS 1 by default)3. LB Model Select LB-S model. 200 Mbps of bandwidth is more than enough.4. LB Type Select Private Load Balancer 5. Finalise	 Click on "Create Load Balancer"Copy code

2 . Linking your application to PrivateLB

To use Private LB, we no longer need to use Ingress.

Traffic is sent directly to the application's service in a private flow !

Scaleway Private LB ⇒ SVC K8s ⇒ Application

In our Service K8s object, we use the following annotations:

service.beta.kubernetes.io/scw-loadbalancer-externally-managed : true service.beta.kubernetes.io/scw-loadbalancer-id : zone/lb_idservice.beta.kubernetes.io/scw-loadbalancer-zone : zoneCopy code

Please just note what CCM will or won’t manage in this case:
• Won’t create/delete the LB.
• Ignores the global configurations (such as size, private mode, IPs).
• Won’t detach private networks attached to the LB.
• won’t manage extra frontends and backends not starting with the service id.
• Will refuse to manage a LB with a name starting with the cluster id.

E.g.:

---apiVersion: v1kind: Servicemetadata: name: test-lb-private namespace: ops-tools annotations:   service.beta.kubernetes.io/scw-loadbalancer-externally-managed: "true"   service.beta.kubernetes.io/scw-loadbalancer-id: "fr-par-1/0d1b714f-2767-4937-a9e5-4399cfd45338"   service.beta.kubernetes.io/scw-loadbalancer-zone: fr-par-1 labels:   app.kubernetes.io/name: test-lb-private   app.kubernetes.io/instance: test-lb-privatespec: ports:   - name: https     port: 443 <= IMPORTANT: 443 port for use of HTTPS certificate in Private LB     protocol: TCP     targetPort: 80   - name: test     port: 81     protocol: TCP     targetPort: 81 selector:   app.kubernetes.io/name: test-lb-private   app.kubernetes.io/instance: test-lb-private type: LoadBalancer <= IMPORTANT: LoadBalancer type for linkCopy code

Apply.

3. Adjust LB settings

Change the name of the frontend using port 443 to the domain you want to use it on.
the “golem.ai” extension is mandatory.

In the following, we'll show you how to use a different domain extension to access the application from a different environment.

If we are accessing the application from a different environment(s) than the one it was created in, we must add the private networks:

4 . Adding a DNS entry

For HTTPS access to the application from its environment or from other environments, we need to add an A entry.

Private @IP for test.golem.ai : 172.16.0.53 (See the loadbalancer's private network @IP)

5 . Create Let's Encrypt certificate and associate it with frontend

Scaleway's Private Loadbalancer doesn’t support automatic management of let's encrypt certificates by CertManager.

So we’ve to manage their entire lifecycle and deploy them ourselves.

To avoid this, we've set up a mechanism using the APIs of Scaleway, CloudFlare and Certbot, as described below.

Requirements :

⇒ Available on the Golem.ai Github Repository : https://github.com/golem-ai/privatelb-scaleway

To use this script, you'll need certbot and the certbot-dns-cloudflare plugin installed on the runtime environment.

Two files :

• certmanager-privatelbscw.sh for LB certificate management
• cloudflare.ini for Certbot ACME management on CloudFlare DNS Provider

Important fields are here :

On certmanager-privatelbscw.sh

##Certbot##lets_server=https://acme-v02.api.letsencrypt.org/directoryauth_email=XXXXXXXXXcloudflare_config=cloudflare.ini##Scaleway API Token##scw_token=XXXXXXXXX##URI : Application name without domain extension##hostname=("test")##Domain extension : golem.ai , toto.ai , tata.ai , titi.ai##default_envs=("golem.ai")##Multi Private Network : To access resources from an environment other than the one from which the SVC K8s was created##multipns=("no")##LBs###test:0d1b714f-2767-4937-a9e5-4399cfd45338private_lbs=("0d1b714f-2767-4937-a9e5-4399cfd45338")zones=("fr-par-1")Copy code

lets_server : Fill in the Let's encrypt environment with which you wish to generate the certificate (production or staging).

auth_email : Fill in the email address we wish to use for certificate generation

scw_token : Secret token for Scaleway API authentication

domains : Fill in the domain we created earlier (test)

multipns : Would we like to access our applications from a non-production environments ? (no / yes)

private_lbs : Our Private LB ID

Zone : Area from which we have created our Private LB

On cloudflare.ini

# Cloudflare API token used by Certbotdns_cloudflare_api_token = XXXXXXXXXCopy code

dns_cloudflare_api_token : Fill with Cloudflare API token

Execution :

The information entered in the example will be used to access the test application via the url test.golem.ai from the production environment.

Once completed, save and run the script!

./certmanager-privatelbscw.sh xxxxxxxx.golem.aiissuer=C = US, O = Let's Encrypt, CN = R3Aug 20 13:34:06 2024 GMT----xxxxxxxx.golem.aiissuer=C = US, O = Let's Encrypt, CN = R3Aug 25 10:56:11 2024 GMT----xxxxxxxx.golem.aiissuer=C = US, O = Let's Encrypt, CN = R3Aug 25 10:47:04 2024 GMT----xxxxxxxx.golem.aiissuer=C = US, O = Let's Encrypt, CN = R3Aug 27 11:42:58 2024 GMT----xxxxxxxx.golem.aiissuer=C = US, O = Let's Encrypt, CN = R3Aug 27 11:48:00 2024 GMT----Could not read certificate from <stdin>40D7FB7F107F0000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:../crypto/store/store_result.c:151:Unable to load certificatetest.golem.ai----Saving debug log to /var/log/letsencrypt/letsencrypt.logRequesting a certificate for test.golem.aiSuccessfully received certificate.Certificate is saved at: /etc/letsencrypt/live/test.golem.ai/fullchain.pemKey is saved at:         /etc/letsencrypt/live/test.golem.ai/privkey.pemThis certificate expires on 2024-08-28.These files will be updated when the certificate renews.Certbot has set up a scheduled task to automatically renew this certificate in the background.- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -If you like Certbot, please consider supporting our work by: * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate * Donating to EFF:                    https://eff.org/donate-le- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100  4887  100  1178  100  3709   1593   5016 --:--:-- --:--:-- --:--:--  6612  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100  4442    0  4442    0     0  10888      0 --:--:-- --:--:-- --:--:-- 10887b33097cf-4e49-456b-990b-1bc8a64caef3fd7c47b7-9214-4945-9bae-67f18bfe0070443{"id":"fd7c47b7-9214-4945-9bae-67f18bfe0070", "name":"test.golem.ai", "inbound_port":443, "backend":{"id":"b33097cf-4e49-456b-990b-1bc8a64caef3", "name":"915d146e-fef6-4873-92f9-58cbea07ac4e_tcp_30715", "forward_protocol":"tcp", "forward_port":30715, "forward_port_algorithm":"roundrobin", "sticky_sessions":"none", "sticky_sessions_cookie_name":"", "health_check":{"port":30715, "check_delay":5000, "check_timeout":5000, "check_max_retries":5, "check_send_proxy":false, "transient_check_delay":null, "tcp_config":{}}, "pool":["172.16.0.54", "172.16.0.7", "172.16.0.25", "172.16.0.9", "172.16.0.21", "172.16.0.18", "172.16.0.15", "172.16.0.31", "172.16.0.14", "172.16.0.17", "172.16.0.22", "172.16.0.27"], "lb":{"id":"0d1b714f-2767-4937-a9e5-4399cfd45338", "name":"private-lb-prod-test", "description":"", "status":"ready", "instances":[], "organization_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "project_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "ip":[], "tags":[], "frontend_count":2, "backend_count":2, "type":"lb-s", "subscriber":null, "ssl_compatibility_level":"ssl_compatibility_level_intermediate", "created_at":"2024-05-30T10:43:08.836695Z", "updated_at":"2024-05-30T12:57:03.854798Z", "private_network_count":1, "route_count":0, "region":"fr-par", "zone":"fr-par-1"}, "send_proxy_v2":false, "timeout_server":10000, "timeout_connect":600000, "timeout_tunnel":600000, "on_marked_down_action":"on_marked_down_action_none", "proxy_protocol":"proxy_protocol_none", "created_at":"2024-05-30T11:39:17.384840Z", "updated_at":"2024-05-30T11:39:17.384840Z", "failover_host":null, "ssl_bridging":false, "ignore_ssl_server_verify":null, "redispatch_attempt_count":0, "max_retries":3, "max_connections":null, "timeout_queue":null}, "lb":{"id":"0d1b714f-2767-4937-a9e5-4399cfd45338", "name":"private-lb-prod-test", "description":"", "status":"ready", "instances":[{"id":"9bf4b0ab-8fcd-405a-9c9c-9e246e03b057", "status":"pending", "ip_address":"", "created_at":"2024-05-30T10:23:15.724826Z", "updated_at":"2024-05-30T12:59:23.088049484Z", "region":"fr-par", "zone":"fr-par-1"}], "organization_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "project_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "ip":[], "tags":[], "frontend_count":2, "backend_count":2, "type":"lb-s", "subscriber":null, "ssl_compatibility_level":"ssl_compatibility_level_intermediate", "created_at":"2024-05-30T10:43:08.836695Z", "updated_at":"2024-05-30T12:57:03.854798Z", "private_network_count":1, "route_count":0, "region":"fr-par", "zone":"fr-par-1"}, "timeout_client":600000, "certificate":{"id":"aba65e0d-7391-4685-b8c3-f86b88c2ade5", "type":"custom", "status":"ready", "common_name":"test.golem.ai", "subject_alternative_name":[], "fingerprint":"", "not_valid_before":"2024-05-30T11:59:18Z", "not_valid_after":"2024-08-28T11:59:17Z", "lb":{"id":"0d1b714f-2767-4937-a9e5-4399cfd45338", "name":"private-lb-prod-test", "description":"", "status":"ready", "instances":[], "organization_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "project_id":"a7ec9296-de32-44d3-9f95-611cd8ee8e20", "ip":[], "tags":[], "frontend_count":2, "backend_count":2, "type":"lb-s", "subscriber":null, "ssl_compatibility_level":"ssl_compatibility_level_intermediate", "created_at":"2024-05-30T10:43:08.836695Z", "updated_at":"2024-05-30T12:57:03.854798Z", "private_network_count":1, "route_count":0, "region":"fr-par", "zone":"fr-par-1"}, "name":"test.golem.ai-2024-05-30", "created_at":"2024-05-30T12:59:21.483814Z", "updated_at":"2024-05-30T12:59:23.049179683Z", "status_details":null}, "certificate_ids":["aba65e0d-7391-4685-b8c3-f86b88c2ade5"], "created_at":"2024-05-30T11:39:17.735956Z", "updated_at":"2024-05-30T12:59:23.045618828Z", "enable_http3":true}Copy code

The certificate has been created and is associated with the frontend on which we want to access the application on port 443.

Renewal only within 30 days of expiry.

Access to the application :

⇒ https://test.golem.ai/

Once again, test.golem.ai is only accessible from our ZeroTrust Golem.ai Private Network.
Outside this network, the application doesn’t exist (172.16.0.53 / non-routable IP address).

nslookup test.golem.aiServer:		127.0.0.53Address:	127.0.0.53#53Non-authoritative answer:Name:	test.golem.aiAddress: 172.16.0.53Copy code