How to execute complex startup commands using Serverless Jobs
Scaleway Serverless Jobs allows you to execute specific startup commands when running a job. Due to technical specifications, complex commands (e.g. piped commands, xargs
commands) may fail occasionally, preventing jobs from running successfully.
You can bypass this limitation by passing complex commands and scripts to a Serverless Job via a secret reference. You can then inject this secret as a file in your job, and call it at startup to execute the commands it contains.
Before you start
To complete the actions presented below, you must have:
- A Scaleway account logged into the console
- Owner status or IAM permissions allowing you to perform actions in the intended Organization
- Created a Serverless Job
How to create a secret containing your command
-
Click Secret Manager in the Security & Identity section of the Scaleway console side menu.
-
In the Region drop-down, select the region in which you want to store your secret. Secrets cannot be moved from one region to another after creation.
-
Click + Create secret.
-
Select Import secret, choose the Opaque secret type, then drag and drop your file to the dedicated area. The maximum file size for your secret is 64 KiB.
-
Choose the Scaleway-managed encryption key, as it requires no configuration on your side.
-
Choose a path for your secret.
-
Enter a name for your secret, and, optionally, add a description and tags.
-
Click Create secret to confirm.
Your file can now be passed to your Serverless Job as a secret reference.
Create a Serverless Job referencing your command as a secret
-
Click Jobs in the Serverless section of the side menu. The jobs page displays.
-
Click Create job. The job creation wizard displays.
-
Select the external container registry.
-
Enter the following image URL:
scaleway/cli:latest
-
Enter a name, select the desired region, and choose the smallest resources available.
-
From the Data tab, add your command file as a secrets reference. Refer to the dedicated documentation on secrets for more information.
-
From the Execution tab, add the following startup command to call your file:
bash /complex_command.sh start
-
Click Create job to finish.
You job is now ready to run.
Complex commands examples
Below are examples of commands that must be passed via a secret referenced as a file in Serverless Jobs. You can find more complex command examples on the Scaleway CLI repository.
Retrieve a specific field from the output using jq
## Retrieve all available instance type names
# Using jq
scw -o json instance server-type list | jq -r '.[].name'
# Using CLI templates
scw instance server-type list -o template="{{ .Name }}"
Filter command output using jq
# Retrieve all available instance type with GPUs
scw -o json instance server-type list | jq '.[] | select (.gpu > 0)'
Parallelize actions using xargs
## Reboot all listed servers at the same time, 8 max concurrent actions
# Using jq
scw -o json instance server list | jq -r '.[].id' | xargs -L1 -P8 scw instance server reboot
# Using CLI templates
scw instance server list -o template="{{ .ID }}" | xargs -L1 -P8 scw instance server reboot
Create arguments for a CLI command and use it in xargs
## List private-nics of all listed servers
# Using jq
scw -o json instance server list | jq -r '.[] | "server-id=\(.id)"' | xargs -L1 scw instance private-nic list
# Using CLI templates
scw instance server list -o template="server-id={{ .ID }}" | xargs -L1 scw instance private-nic list