Skip to navigationSkip to main contentSkip to footerScaleway DocsSparklesIconAsk our AI
SparklesIconAsk our AI

How to create a workflow using Data Orchestrator

InformationOutlineIcon
Note

Data Orchestrator is currently in Private Beta. During this Beta period:

  • Code mode is the only supported mode.
  • The workflows only run Scaleway Serverless Jobs.

Before you start

To complete the actions presented below, you must have:

Create a workflow definition YAML file

Data Orchestrator requires the user to upload a YAML file to use as a workflow definition.

CheckCircleOutlineIcon
Tip

This is a basic example of a sequential workflow, more examples will be available soon.

  1. Create a YAML file, copy/paste the following content into the file.
    document:
      dsl: 1.0.0
      namespace: <namespace_name>
      name: <workflow_name>
      version: 1.0.0
    do:
    - task1:
        call: "serverless_job"
        metadata:
          description: "The first job"
        with:
          id: "<Serverless_job1_ID>"
          region: "fr-par"
    - task2:
        call: "serverless_job"
        metadata:
          description: "The second job"
        with:
          id: "<Serverless_job2_ID>"
          region: "fr-par"
    - task3:
        call: "serverless_job"
        metadata:
          description: "The third job"
        with:
          id: "<Serverless_job3_ID>"
          region: "fr-par"
  2. Replace the ID placeholders with the IDs of your Serverless Jobs. You can copy them from the Serverless Jobs page.
    InformationOutlineIcon
    Note

    The fields namespace and name are required. They can be used as metadata to help organize your YAML files.

  3. Save the YAML file.

Create a Data Orchestrator workflow

  1. Click Data Orchestrator in the Data & Analytics section of the console side menu. The list of your workflows displays.
  2. Click + Create workflow. The Data Orchestrator workflow creation wizard displays.
  3. Select the region in which your workflow will be created.
  4. Enter a name or use the automatically generated one.
  5. Enter a description (optional)
  6. Click Create workflow only to finish, or click Create workflow and add tasks if you want to upload a workflow next.

Upload the workflow definition

The workflow upload window is displayed during the creation of a workflow or when clicking Edit from a workflow in Code view.

  1. Click Upload.
  2. Select the YAML file to use as the workflow definition.
  3. Click Save changes to finish, or click Save and run if you want to save and run the workflow.

Run a workflow

  1. Click Data Orchestrator in the Data & Analytics section of the console side menu. The list of your workflows displays.
  2. Click the name of the workflow you want to run. The workflow Overview tab displays.
  3. Click the Actions button, then click Run in the drop-down menu. Your workflow starts, and a new entry appears in the Runs tab.
CheckCircleOutlineIcon
Tip

On the Data Orchestrator page, you can also click the run icon next to the workflow you want to run.

SearchIcon
No Results