---
title: Set up a serverless architecture for handling large messages using Scaleway's NATS, Serverless Functions, and Object Storage.
description: Learn how to build a serverless architecture for handling large messages with Scaleway's NATS, Serverless Functions, and Object Storage. Follow our step-by-step Terraform/OpenTofu-based tutorial for asynchronous file conversion using NATS, functions, and triggers.
products:
  - nats
  - functions
  - object-storage
tags: messaging nats terraform big-data
dates:
  validation: 2025-05-19
  posted: 2024-01-22
  validation_frequency: 12
difficulty: beginner
usecase:
  - best-practices
ecosystem:
  - scaleway-only
---
import image from './assets/scaleway-large-messages-architecture.webp'

import Requirements from '@macros/iam/requirements.mdx'


In this tutorial, we will build a simple architecture to store and automatically convert images to PDF. The focus is on transferring large messages using a messaging service, specifically the [Scaleway NATS](/nats/quickstart/) server. Our setup involves using the [Object Storage](/object-storage/quickstart/) service for image storage and the [Serverless Functions](/serverless-functions/quickstart/) service for conversion.

We show how to provision all the required Scaleway resources via Terraform/OpenTofu, but you can also use the [console](https://account.scaleway.com/login), the [API](https://www.scaleway.com/en/developers/api/) or one of our other supported [developer tools](https://www.scaleway.com/en/developers/).

You can retrieve the full source code in [this repository](https://github.com/rouche-q/serverless-examples/tree/main/projects/large-messages/README.md), so you can choose to either jump straight into the code or else follow along with the step-by-step explanation below to create the architecture yourself.

<Requirements />

- A Scaleway account logged into the [console](https://console.scaleway.com)
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
- Installed [Terraform/OpenTofu](/terraform/quickstart/) on your local machine
- [Set up authentication for the Terraform/OpenTofu provider](https://registry.terraform.io/providers/scaleway/scaleway/latest/docs#authentication)
- Installed [Docker](/tutorials/install-docker-ubuntu-bionic/) on your local machine
- Installed [NATS CLI](https://github.com/nats-io/natscli) on your local machine
- [AWS CLI configured with Scaleway credentials](/object-storage/api-cli/object-storage-aws-cli/#how-to-install-the-aws-cli)

## Architecture

<Lightbox image={image} alt="" />

This tutorial's architecture is very simple, consisting of two main components:

- A producer implemented as a shell script responsible for uploading an image to the Object Storage bucket and sending a message to the NATS server.
- A consumer implemented as a serverless Python function designed to receive the message, parse it, extract the image URL, download it from the bucket, convert it to a PDF, and then re-upload the resulting PDF.

Three essential services are required to ensure everything is working together:

- An Object Storage bucket for storing the image file.
- A Scaleway NATS server responsible for receiving and dispatching messages sent by our producer.
- A triggering mechanism in NATS that forwards the message to our consumer.

## Prepare the infrastructure

1. Create a `main.tf` file, which we will use to provision all our Scaleway resources and infrastructure. Paste the following code to import the necessary providers.
    <Message type="tip">
    Remember that you can refer to the [code repository](https://github.com/rouche-q/serverless-examples/tree/main/projects/large-messages/README.md) to check all code files.
    </Message>
    ```hcl
    terraform {
      required_providers {
        scaleway = {
          source = "scaleway/scaleway"
        }
        null = {
          source = "hashicorp/null"
        }
        random = {
          source = "hashicorp/random"
        }
        archive = {
          source = "hashicorp/archive"
        }
      }
      required_version = ">= 0.13"
    }
    ```
    The Scaleway provider is needed, but also three providers from HashiCorp that we will use later in the tutorial.

2. Include two variables to enable the secure passage of your Scaleway credentials. Then initialize the Scaleway provider in the `fr-par-1` region.
    ```hcl
    variable "scw_access_key_id" {
      type      = string
      sensitive = true
    }

    variable "scw_secret_access_key" {
      type      = string
      sensitive = true
    }

    provider "scaleway" {
      zone = "fr-par-1"
    }
    ```

3. Create a `terraform.tfvars` file and add your Scaleway access key and access key ID to it.
    ```terraform
    scw_access_key_id     = "YOUR_SCW_SECRET_KEY_ID"
    scw_secret_access_key = "YOUR_SCW_SECRET_KEY"
    ```

4. Continuing in the `main.tf` file, add the following Terraform/OpenTofu code to create an Object Storage bucket that will be used for storing your images.
    ```hcl
    resource "random_id" "bucket" {
      byte_length = 8
    }

    resource "scaleway_object_bucket" "large_messages" {
      name = "large-messages-${random_id.bucket.hex}"
    }

    resource "scaleway_object_bucket_acl" "large_messages" {
      bucket = scaleway_object_bucket.large_messages.id
      acl    = "private"
    }

    output "bucket_name" {
      value       = scaleway_object_bucket.large_messages.name
      description = "Bucket name to use with the producer script"
    }
    ```
    In this code, the resource `random_id.bucket` generates a random ID, which is then passed to the object bucket to ensure its uniqueness. Additionally, a `scaleway_object_bucket_acl` ACL is applied to the bucket, setting it to private and outputting the bucket name for use in your producer.

5. Add these resources to create a NATS account and your NATS credentials file:
    ```hcl
    resource "scaleway_mnq_nats_account" "large_messages" {
      name = "nats-acc-large-messages"
    }

    resource "scaleway_mnq_nats_credentials" "large_messages" {
      name       = "nats-large-messages-creds"
      account_id = scaleway_mnq_nats_account.large_messages.id
    }

    resource "local_file" "nats_credential" {
      content         = scaleway_mnq_nats_credentials.large_messages.file
      filename        = "large-messages.creds"
      file_permission = 644
    }

    output "nats_url" {
      value       = scaleway_mnq_nats_account.large_messages.endpoint
      description = "NATS url to use with the producer script"
    }
    ```
    We also output the NATS server URL.

6. Run `terraform init` and `terraform apply` to create the resources in your Scaleway account.

## Create the producer

As mentioned earlier, the producer will be implemented as a straightforward shell script.

1. Create a file named `upload_img.sh`. Inside it, define two variables using Terraform/OpenTofu commands to retrieve the bucket name and the NATS URL.
    ```bash
    SCW_BUCKET="$(terraform output bucket_name)"
    SCW_NATS_URL="$(terraform output nats_url)"
    ```

2. Configure the NATS CLI with the NATS URL and the credentials file.
    ```bash
    nats context save large-messages --server=$SCW_NATS_URL --creds=./large-messages.creds
    nats context select large-messages
    ```
    Our script takes the file path that we want to upload as the first parameter.

    To upload the file, we will use the AWS CLI configured with the Scaleway endpoint and credentials because Scaleway Object storage is fully compliant with S3.

3. Pass the path to the AWS CLI command as follows:
    ```bash
    aws s3 cp $1 s3://$SCW_BUCKET
    ```

4. Paste the following command into the script to have it send a message to your NATS server, including the name of the uploaded file.
    ```bash
    nats pub large-messages $(basename $1)
    ```

## Create the consumer

We continue using the Scaleway ecosystem and deploy the consumer using a Serverless Function in Python.
1. Create a folder `function`. Inside the folder, create another folder `handler` with a file named `large_messages.py`.
2. In addition to the `handler` folder, create a `requirements.txt` file. Your directory structure should resemble the following:
    You should have a structure like the following:
    ```
    function/
    ├── handler/
    │   └── large_messages.py
    └── requirements.txt
    ```

3. In the `requirements.txt` add:
    ```
    boto3
    img2pdf
    ```
    Within the serverless function, we use two Python libraries. First, `boto3` is configured similarly to AWS CLI, enabling the download and upload of files to the bucket. Second, `img2pdf` is used to convert our image into a PDF format.

4. Next create the code of your function. In the file `large_messages.py`, import the libraries that we need and create an empty handler function.
    ```python
    import os
    import boto3
    from botocore.exceptions import ClientError
    import img2pdf
    from PIL import Image

    def handle(event, context):
        return {
            "body": {
                "message": "Hello world!"
            },
            "statusCode": 200
        }
    ```

5. Before proceeding with the function's logic, improve the Terraform/OpenTofu code by adding the following code to your `main.tf` file:
    ```hcl
    resource "null_resource" "install_dependencies" {
      provisioner "local-exec" {
        command = <<-EOT
          cd function
          [ -d "./function/package" ] && rm -rf ./package
          PYTHON_VERSION=3.11
          docker run --rm -v $(pwd):/home/app/function --workdir /home/app/function rg.fr-par.scw.cloud/scwfunctionsruntimes-public/python-dep:$PYTHON_VERSION \
           pip3 install --upgrade -r requirements.txt --no-cache-dir --target ./package
          cd ..
        EOT
      }

      triggers = {
        hash = filesha256("./function/handler/large_messages.py")
      }
    }

    data "archive_file" "function_zip" {
      type        = "zip"
      source_dir  = "./function"
      output_path = "./function.zip"

      depends_on = [null_resource.install_dependencies]
    }
    ```
    The `null_resource` is used to download and package the correct versions of the libraries that we use with the function. Refer to the [documentation on Python dependencies](/serverless-functions/how-to/package-function-dependencies-in-zip/#dependencies) for more information.

6. Create the function namespace.
    ```hcl
    resource "scaleway_function_namespace" "large_messages" {
      name        = "large-messages-function"
      description = "Large messages namespace"
    }
    ```

7. Add the resource to set up the function.
    ```hcl
    resource "scaleway_function" "large_messages" {
      namespace_id = scaleway_function_namespace.large_messages.id
      runtime      = "python311"
      handler      = "handler/large_messages.handle"
      privacy      = "private"
      zip_file     = "function.zip"
      zip_hash     = data.archive_file.function_zip.output_sha256
      deploy       = true
      memory_limit = "2048"
      environment_variables = {
        ENDPOINT_URL  = scaleway_object_bucket.large_messages.api_endpoint
        BUCKET_REGION = scaleway_object_bucket.large_messages.region
        BUCKET_NAME   = scaleway_object_bucket.large_messages.name
      }
      secret_environment_variables = {
        ACCESS_KEY_ID     = var.scw_access_key_id
        SECRET_ACCESS_KEY = var.scw_secret_access_key
      }

      depends_on = [data.archive_file.function_zip]
    }
    ```
    The code sets up the function and connects it to the namespace. It tells the system to use Python version 3.11 and specifies which part of the Python file should run (the handler). It also adds important information, like environment variables and secrets, that our function will need to work correctly.
    Essential environment variables and secrets to use in our function logic are also added.

8. Create the function trigger to "wake up" the function when a NATS message comes in.
    ```hcl
    resource "scaleway_function_trigger" "large_messages" {
      function_id = scaleway_function.large_messages.id
      name        = "large-messages-trigger"
      nats {
        account_id = scaleway_mnq_nats_account.large_messages.id
      subject    = "large-messages"
      }
    }
    ```
    It defines which account ID and subject to observe for getting messages.

9. Go back to `large_messages.py` and get the environment variables outside the `hamdler` function.
    ```python
    endpoint_url = os.getenv("ENDPOINT_URL")
    bucket_region = os.getenv("BUCKET_REGION")
    bucket_name = os.getenv("BUCKET_NAME")
    access_key_id = os.getenv("ACCESS_KEY_ID")
    secret_access_key = os.getenv("SECRET_ACCESS_KEY")
    ```

10. Get the input file name from the body, define the PDF file name from this, and set up the Amazon S3 client to upload the file with Scaleway credentials.
    ```python
    input_file = event['body']
    output_file = os.path.splitext(input_file)[0] + ".pdf"
    s3 = boto3.client('s3', endpoint_url=endpoint_url,
                      region_name=bucket_region,
                      aws_access_key_id=access_key_id,
                      aws_secret_access_key=secret_access_key
    ```

11. Outside the `handle` function, create a new function `convert_img_to_pdf`.
    ```python
    def convert_img_to_pdf(img_path, pdf_path):
        image = Image.open(img_path)
        pdf_bytes = img2pdf.convert(image.filename)
        file = open(pdf_path, "wb")
        file.write(pdf_bytes)
        image.close()
        file.close()
        print("Successfully made pdf file")
    ```

12. Download the image from the bucket using the Amazon S3 client.
    ```python
    s3.download_file(bucket_name, input_file, input_file)
    print("Object " + input_file + " downloaded")
    ```

13. Convert the image with the dedicated function and reupload it in the bucket.
    ```python
    convert_img_to_pdf(input_file, output_file)
    s3.upload_file(output_file, bucket_name, output_file)
    print("Object " + input_file + " uploaded")
    ```

14. Put a `try/except` around the code to gracefully handle any errors coming from the Object Storage client.
    ```python
    try:
        s3.download_file(bucket_name, input_file, input_file)
        print("Object " + input_file + " downloaded")
        convert_img_to_pdf(input_file, output_file)
        s3.upload_file(output_file, bucket_name, output_file)
        print("Object " + input_file + " uploaded")
    except ClientError as e:
        print(e)
      return {
            "body": {
               "message": e.response['Error']['Message']
            },
            "statusCode": e.response['Error']['Code']
        }
    ```

## Test your work

All the infrastructure and the code of your function are ready, and you can now test your converter.
```bash
# Make sure your infrastructure is provisioned
terraform init
terraform apply

# Use your producer script
./upload_img.sh test.png #It can be any type of image
```

## Conclusion, going further

In this introductory tutorial, we have demonstrated the usage of the Scaleway NATS server, along with other services from the Scaleway ecosystem, to facilitate the transfer of large messages surpassing the typical size constraints. There are possibilities to expand upon this tutorial for various use cases, such as:

	- Extending the conversion capabilities to handle different document types like `docx`.
	- Sending URLs directly to NATS and converting HTML content to PDF.