Data Orchestrator - Quickstart
Data Orchestrator is designed to help the user automate, schedule, and manage data workflows across various systems and environments. Follow this guide to create your first workflow and take it through its full lifecycle.
Before you start
To complete the actions presented below, you must have:
- A Scaleway account logged into the console
- Owner status or IAM permissions allowing you to perform actions in the intended Organization
- A Container Registry namespace
- An image pushed to your namespace (for task execution)
- Created at least three Serverless Jobs
- Basic familiarity with YAML syntax
- Basic familiarity with Serverless workflow.io language
Create a workflow definition YAML file
Data Orchestrator requires the user to upload a YAML file to use as a workflow definition.
- Create a YAML file, copy and paste the following content into the file.
document: dsl: 1.0.0 namespace: <namespace_name> name: <workflow_name> version: 1.0.0 do: - task1: call: "serverless_job" metadata: description: "The first job" with: id: "<Serverless_job1_ID>" region: "fr-par" - task2: call: "serverless_job" metadata: description: "The second job" with: id: "<Serverless_job2_ID>" region: "fr-par" - task3: call: "serverless_job" metadata: description: "The third job" with: id: "<Serverless_job3_ID>" region: "fr-par" - Replace the ID placeholders with the IDs of your Serverless Jobs. You can copy them from the Serverless Jobs page of the console.
- Save the YAML file.
Create a Data Orchestrator workflow
- Click Data Orchestrator in the Data & Analytics section of the console side menu. The list of your workflows displays.
- Click + Create workflow. The Data Orchestrator workflow creation wizard displays.
- Select the region in which your workflow will be created.
- Enter a name or use the automatically generated one.
- Enter a description (optional).
- Click Create workflow only to finish, or click Create workflow and add tasks to upload a workflow definition file next.
Upload the workflow definition
The workflow upload window is displayed during the creation of a workflow or when clicking Edit from a workflow in Code view.
- Click Upload.
- Select the YAML file to use as the workflow definition.
- Click Save changes to finish, or click Save and run if you want to save and run the workflow.
Run a workflow
- Click Data Orchestrator in the Data & Analytics section of the console side menu. The list of your workflows displays.
- Click the name of the workflow you want to run. The workflow Overview tab displays.
- Click the Actions button, then click Run in the drop-down menu. Your workflow starts, and a new entry appears in the Runs tab.
Monitor a running workflow
- Click the name of the workflow you want to monitor. The workflow Overview tab displays.
- Make sure the last run status is
RunningorQueued. - Click View run next to the last run status. The run information displays.
Schedule a workflow
- Click the name of the workflow you want to manage. The workflow Overview tab displays.
- Click the Settings tab.
- Check the Enable run scheduling box.
- Enter the cron expression and the time zone.
- Click Save changes.
Delete a workflow
- Click the name of the workflow you want to delete. The workflow Overview tab displays.
- Navigate to the Settings tab.
- Click Delete Workflow.
- Type DELETE to confirm.
- Click Delete Workflow.
Still need help?Create a support ticket