import copy

Importing a copy of S3 Objects from one bucket to another

Published on 27 May 2021

Object Storage allows you to import a copy of your objects in a bucket to another bucket using the AWS-CLI.

The sync command is used to synchronize directories and S3 prefixes.

This command will:

Recurringly copy new and updated files from the source directory to the target one
Use the CopyObject APIs to list objects in the source and target buckets and identify which ones they have in common
Compare both to determine disparities in LastModified dates between the same objects

Identity and Access Management (IAM):

If you have activated IAM, you may need certain IAM permissions to carry out some actions described on this page. This means:

you are the Owner of the Scaleway Organization in which the actions will be carried out, or
you are an IAM user of the Organization, with a policy granting you the necessary permission sets

Note:

When applied on a versioned bucket, the sync command copies only the most recent version of the objects.

Requirements:

You have an account and are logged into the Scaleway console
You have configured your API keys
You have created an Object Storage bucket

Importing copied objects from the same project

To import copied objects to a bucket in the same Project, use the aws s3 sync command.

Important:

Before importing a copy, make sure all of the source bucket’s objects are in one of the two classes: STANDARD or ONEZONE_IA (only available in FR-PAR). If an object is in GLACIER class, the copy import will fail. To learn how to edit storage classes, refer to this documentation page.

aws s3 sync s3://$SourceBucket s3://$TargetBucket

Replace $SourceBucket with the name of the bucket where the copy is currently stored, and $TargetBucket with the name of the bucket the copy will be stored in.

Importing copied objects from a different project

To import a copy of objects in a bucket to another bucket in a different Project, you need to implement a bucket policy.

A bucket policy is a resource-based policy option. In this context, bucket policies are used to grant the target Project access to the source bucket.

Important:

Before importing a copy, make sure all of the source bucket’s objects are in one of the two classes: STANDARD or ONEZONE_IA (only available in FR-PAR). If an object is in GLACIER class, the copy import will fail. To learn how to edit storage classes, refer to this documentation page.

Create a file with your bucket policy for the source bucket.

Note:

Bucket policies use a JSON-based access policy language. You can find more details about the JSON policy grammar on this page.

{
    "Version": "2012-10-17",
    "Id":"MyBucketPolicy",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect":"Allow",
            "Principal":{
                "SCW":"project_id:<TARGET_PROJECT_ID>"
            },
            "Action":[
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource":[
                "<BUCKET_NAME>",
                "<BUCKET_NAME>/*"
            ]
        }
    ]
}

Run the following command:

$ aws s3api put-bucket-policy --bucket <SOURCE_BUCKET> --profile <SOURCE_PROJECT> --policy file://bucket-policy.json

If necessary, create the target bucket in the target project.

$ aws s3api create-bucket --bucket <TARGET_BUCKET> --profile <TARGET_PROJECT>
{
  "Location": "/<TARGET_BUCKET>"
}

Copy objects and import using the sync command.
Replace <SOURCE_BUCKET> with the name of the bucket located in the source project and <TARGET_BUCKET> with the name of the bucket in the target project.
```
$ aws s3 sync s3://<SOURCE_BUCKET> s3://<TARGET_BUCKET> --profile <TARGET_PROJECT>
```