Using Proxy Protocol v2 with a Scaleway Load Balancer

Proxy Protocol allows a backend application, like an Apache or Nginx web server, to retrieve client connection information that has passed through a load-balanced infrastructure. The protocol transports connection information, including the originating IP address, the proxy server IP address, and both ports.

This tutorial shows you how and why to enable Proxy Protocol on your Scaleway Load Balancer, and how to configure your backend server application to correctly handle the protocol.

Before you start

To complete the actions presented below, you must have:

• A Scaleway account logged into the console
• Owner status or IAM permissions allowing you to perform actions in the intended Organization
• An SSH key
• Created and configured a Load Balancer

Do I need to enable Proxy Protocol on my Load Balancer?

Without Proxy Protocol enabled, the backend server receives only the Load Balancer's IP address in its access logs, and not the IP address of the original client:

51.159.26.16 - - [28/Jun/2019:13:42:25 +0000] "GET /favicon.ico HTTP/1.1" 200 26066 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
51.159.26.16 - - [28/Jun/2019:13:42:25 +0000] "GET / HTTP/1.1" 200 26099 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
51.159.26.16 - - [28/Jun/2019:13:42:26 +0000] "GET /favicon.ico HTTP/1.1" 200 26068 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
51.159.26.16 - - [28/Jun/2019:13:42:49 +0000] "GET / HTTP/1.1" 200 26022 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"

In the example above, the IP 51.159.26.16 is visible in the log file, which is the Load Balancer's IP address and not the IP of the different users.

With Proxy Protocol enabled, an additional header is added, showing the proxy (Load Balancer) IP address, but also the client IP address and source/destination ports:

PROXY TCP4 51.159.26.16 203.0.113.1 12345 80
- - [28/Jun/2019:13:42:25 +0000] "GET /favicon.ico HTTP/1.1" 200 26066 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

PROXY TCP4 51.159.26.16 203.0.113.1 12345 80
- - [28/Jun/2019:13:42:25 +0000] "GET / HTTP/1.1" 200 26099 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

PROXY TCP4 51.159.26.16 203.0.113.1 12345 80
- - [28/Jun/2019:13:42:26 +0000] "GET /favicon.ico HTTP/1.1" 200 26068 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

PROXY TCP4 51.159.26.16 203.0.113.1 12345 80
- - [28/Jun/2019:13:42:49 +0000] "GET / HTTP/1.1" 200 26022 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"

In the example above, the IP 51.159.26.16 represents the Load Balancer's IP, 203.0.113.1 is the original client's IP, and 12345 and 80 are the source and destination ports.

This additional information can be useful for the delivery of localized content, blacklisting of abusive users, or logging purposes. If these use cases do not apply to you, it may not be useful to enable Proxy Protocol.

Note that Proxy Protocol is more commonly activated for Load Balancers using TCP protocol. Load Balancers using HTTP protocol already pass information about the client IP address to the backend servers via an HTTP X-Forwarded-For header, without needing to activate Proxy Protocol. If your Load Balancer uses HTTP protocol and you do not require the standardized information in the Proxy Protocol headers at the backend server, the X-Forwarded-For headers may be sufficient.

Enabling Proxy Protocol on your Load Balancer

You can enable Proxy Protocol for your Load Balancer when you create your Load Balancer's backend, or after creation by editing a backend. Proxy protocol is one of the basic backend settings you are prompted to configure.

See our advanced documentation on configuring backends for full information about the different Proxy Protocol versions available, and the configuration process.

Configuring your backend web server for Proxy Protocol

Once you have enabled Proxy Protocol on your Load Balancer, you must configure your backend web server to handle this information. The sections below show how to configure some common backend web server applications for Proxy Protocol:

Configuring Proxy Protocol in Nginx Web Server

The proxy_protocol parameter must be set within the http {} block of the listen directive of a server block to configure NGINX to accept Proxy protocol headers.

1. Make sure that Nginx is installed with the http_realip_module. This is the case in the precompiled version that is delivered with Ubuntu Bionic Beaver (18.04).

   nginx -V 2>&1 | grep -- 'http_realip_module'

2. Open the configuration file of nginx, i.e. /etc/nginx/nginx.conf in a text editor, for example, nano:

   nano /etc/nginx/nginx.conf

3. Enable the proxy_protocol by adding/modifying the following lines in the server {} block:

   Tip

   Skip forward to step 9 to see what the whole configuration file should look like when you have finished the modifications described here.

   server {
       ...
       listen 80   proxy_protocol;
       listen 443  ssl proxy_protocol;
       ...
       }

4. Set the IP address of the Load Balancer with the set_real_ip_from directive in the server {} block. Make sure you replace {LB_IP} with the IP address of your Load Balancer:

   set_real_ip_from {LB_IP};

5. To change the IP address of the Load Balancer to the clients IP address, received from the PROXY protocol header, specify in the server {} block the proxy_protocol parameter to the real_ip_header directive:

   real_ip_header proxy_protocol;

6. As the client's IP address is now known to Nginx, configure its correct logging. Set the proxy_set_header directive with the $remote_addr variable in the http {} block:

   http {
       proxy_set_header X-Real-IP       $remote_addr;
       proxy_set_header X-Forwarded-For $remote_addr;
   }

7. Configure the $remote_addr variable to the log_format directive in the http block:

   http {
       #...
       log_format logs '$remote_addr - $remote_user [$time_local] '
                           '"$request" $status $body_bytes_sent '
                           '"$http_referer" "$http_user_agent"';
   }

8. Add a log line at the end of the configuration block:

   access_log /var/log/nginx/access.log logs;

9. Check that your configuration file looks like the following example:

   http {
       proxy_set_header X-Real-IP       $remote_addr;
       proxy_set_header X-Forwarded-For $remote_addr;
   
       log_format logs '$remote_addr - $remote_user [$time_local] '
                           '"$request" $status $body_bytes_sent '
                           '"$http_referer" "$http_user_agent"';
       #...
   
       server {
           server_name example.com;
   
           listen 80   proxy_protocol;
   
           set_real_ip_from {LB_IP};
           real_ip_header proxy_protocol;
   
           #...
         }
       access_log /var/log/nginx/access.log logs;
   }

10. Save the configuration file, exit the editor, and test the syntax of the file:

    nginx -t

11. Restart Nginx:

    systemctl restart nginx.service

    195.154.228.158 - - [28/Jun/2019:15:44:23 +0000] "GET /favicon.ico HTTP/1.1" 200 26062 "http://51.159.26.16/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
    51.159.26.16 - - [28/Jun/2019:15:44:31 +0000] "GET /index.php HTTP/1.1" 200 26100 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
    195.154.228.158 - - [28/Jun/2019:15:44:32 +0000] "GET /favicon.ico HTTP/1.1" 200 26065 "http://51.159.26.16/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
    51.159.26.16 - - [28/Jun/2019:15:44:50 +0000] "GET /index.php HTTP/1.1" 200 26097 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

    Important

    Once Proxy Protocol is enabled the Nginx virtual host will no longer accept direct connections. See the NGINX Proxy Protocol documentation for full details.

Configuring Proxy Protocol in Apache web server

• For Apache < 2.4.30, install mod-proxy-protocol
• For Apache >=2.4.30, use mod_remoteip:
  a2enmod remoteip
  RemoteIPProxyProtocol On
  systemctl restart apache2

Configuring Proxy Protocol in Apache with mod_remoteip

1. Enable the module remoteip:
   a2enmod remoteip

2. Configure the Apache virtual host configuration:
   <VirtualHost *:80>
       ...
       RemoteIPProxyProtocol On
       ...
   </VirtualHost>

3. Restart the Apache web server:
   apache2ctl restart

Configuring Kubernetes Ingress

Deploying via kubectl

1. Annotate the ingress controller service:

   kubectl annotate -n ingress-nginx services ingress-nginx-controller \
     "service.beta.kubernetes.io/scw-loadbalancer-proxy-protocol-v2=true"
   kubectl annotate -n ingress-nginx services ingress-nginx-controller \
     "service.beta.kubernetes.io/scw-loadbalancer-use-hostname=true"

2. Patch the ConfigMap:

   kubectl patch -n ingress-nginx configmaps ingress-nginx-controller \
     -p '{"data":{"use-forwarded-headers":"true","compute-full-forwarded-for":"true","use-proxy-protocol":"true"}}'

Deploying via Helm

1. Create nginx-ingress-scw.yaml:

   controller:
     kind: DaemonSet
     service:
       externalTrafficPolicy: "Local"
       annotations:
         service.beta.kubernetes.io/scw-loadbalancer-proxy-protocol-v2: "true"
         service.beta.kubernetes.io/scw-loadbalancer-use-hostname: "true"
     config:
       use-forwarded-headers: "true"
       compute-full-forwarded-for: "true"
       use-proxy-protocol: "true"

2. Install or upgrade the official chart:

   helm upgrade --install nginx-ingress oci://ghcr.io/nginx/charts/nginx-ingress \
     --namespace ingress-nginx --create-namespace \
     -f nginx-ingress-scw.yaml