Encode videos using Serverless Jobs and FFMPEG

This tutorial demonstrates the process of encoding videos retrieved from Object Storage using Serverless Jobs. Media encoding is a resource-intensive task of prolonged duration, making it a highly suitable candidate for Serverless Jobs. The job takes a video file as its input, encodes it using a Docker image based on FFMPEG, then uploads the encoded video back to the Object Storage bucket.

Before you start

To complete the actions presented below, you must have:

• A Scaleway account logged into the console
• Owner status or IAM permissions allowing you to perform actions in the intended Organization
• An Object Storage bucket
• A valid API key
• Installed Docker engine

Creating the job image

The initial step involves defining a Docker image for interacting with the Object Storage using MinIO and performing a video encoding task using FFMPEG.

1. Create a bash script encode.sh with the following content:

   #!/bin/sh
   set -e
   
   echo "Configuring Object Storage access for MinIO"
   mc config host add scw "https://$JOB_S3_ENDPOINT/" "$JOB_S3_ACCESS_KEY" "$JOB_S3_SECRET_KEY"
   
   echo "Downloading the file from S3"
   mc cp "scw/$JOB_INPUT_PATH/$JOB_INPUT_FILENAME" "/tmp/$JOB_INPUT_FILENAME"
   
   echo "Encoding the file"
   ffmpeg -i "/tmp/$JOB_INPUT_FILENAME" -vcodec libx264 -acodec aac "/tmp/$JOB_OUTPUT_FILENAME"
   
   echo "Uploading the encoded file to the S3"
   mc cp "/tmp/$JOB_OUTPUT_FILENAME" "scw/$JOB_OUTPUT_PATH/$JOB_OUTPUT_FILENAME"

   That bash script downloads a video from an Object Storage bucket, encodes that video using FFMPEG, and then uploads the encoded video into the bucket, by leveraging a couple of environment variables which will be detailed in the following sections.

   Note

   For illustration purposes, this script encodes a video using the x264 video codec and the AAC audio codec. Encoding settings can be modified using command-line parameters to FFMPEG.

2. Use that script as an entrypoint for a new Docker image, by creating the following Dockerfile:

   FROM linuxserver/ffmpeg:amd64-latest
   
   # Install the MinIO client
   RUN curl https://dl.min.io/client/mc/release/linux-amd64/mc -o /usr/local/bin/mc
   RUN chmod +x /usr/local/bin/mc
   
   # Define the image entrypoint
   COPY encode.sh /encode.sh
   RUN chmod +x /encode.sh
   
   ENTRYPOINT /encode.sh

   This Dockerfile uses linuxserver/ffmpeg as a base image bundled with FFMPEG along with a variety of encoding codecs and installs MinIO as a command-line client to copy files over Object Storage.

3. Build and push the image to your Container Registry:

   docker build . -t <registry and image name>
   docker push <registry and image name>

   Note

   You can find the name and endpoint of your Container Registry from the Scaleway console

Creating the serverless job

1. In the Scaleway Console, click Jobs in the Serverless section of the side menu, then Create job to display the job creation wizard.

2. Select the Scaleway Container Registry and your Container Registry namespace from the drop-down list, then select the image from the previous section.

   Open Image over content
   Close current Lightbox

3. Choose your job name, description, and region. In the Resources section, pick a reasonable amount of resources.

   Note

   The allocated CPU defines the speed of the encoding process. The allocated memory depends on the encoding codec and bitrate. For instance, encoding a Full HD video with the x264 codec takes about 3 GB of memory.

4. Toggle the Advanced options section and add 3 environment variables:

   • JOB_S3_ENDPOINT is your Object Storage endpoint (e.g. s3.nl-ams.scw.cloud).
   • JOB_S3_ACCESS_KEY is your API access key.
   • JOB_S3_SECRET_KEY is your API secret key.
   Open Image over content
   Close current Lightbox

5. Leave the other sections empty, then click Create job.

Triggering the serverless job

Ensure that your Object Storage bucket contains at least one video that can be encoded.

1. In the Scaleway Console, go to Serverless Jobs and click on the name of your job. The job Overview tab displays.

2. Click the Actions button, then click Run job with options in the drop-down menu.

3. Add 4 environment variables:

   • JOB_INPUT_PATH is the folder containing the video to encode, including your Object Storage bucket name.
   • JOB_INPUT_FILENAME is the file name of the video to encode, including the file extension.
   • JOB_OUTPUT_PATH is the folder containing the encoded video that will be uploaded, including your Object Storage bucket name.
   • JOB_OUTPUT_FILENAME is the file name of the encoded video that will be uploaded.
   Open Image over content
   Close current Lightbox

4. Click Run job.

The progress and details for your Job run can be viewed in the Job runs section of the job Overview tab in the Scaleway console. You can also access the detailed logs of your job in Cockpit.

Once the run status is Succeeded, the encoded video can be found in your Object Storage bucket under the folder and file name specified above in the environment variables.

curl -X POST \
  -H "X-Auth-Token: <API Key>" \
  -H "Content-Type: application/json" \
  -d '{ "environment_variables": { "JOB_INPUT_FILENAME": "...", "JOB_INPUT_PATH" : "...", "JOB_OUTPUT_FILENAME" : "...", "JOB_OUTPUT_PATH" : "..." } }' \
  "https://api.scaleway.com/serverless-jobs/v1alpha1/regions/<Region>/job-definitions/<Job ID>/start"