Importing a copy of S3 Objects from one bucket to another

Reviewed on 28 May 2021 • Published on 27 May 2021

Object Storage allows you to import a copy of your objects in a bucket to another bucket using the AWS-CLI.

The sync command is used to synchronize directories and S3 prefixes.

This command will:

Recurringly copy new and updated files from the source directory to the target one
Use the CopyObject APIs to list objects in the source and target buckets and identify which ones they have in common
Compare both to determine disparities in LastModified dates between the same objects

Note:

When applied on a versioned bucket, the sync command copies only the most recent version of the objects.

Requirements:

You have an account and are logged into the Scaleway Console - You have configured your API keys
You have created an Object Storage bucket

Importing copied objects from the same project

To import copied objects to a bucket in the same Project, use the aws s3 sync command.

Important:

Before importing a copy, make sure all of the source bucket’s objects are in STANDARD class. If an object is in GLACIER class, the copy import will fail. To learn how to change storage classes, refer to this documentation page

aws s3 sync s3://$SourceBucket s3://$TargetBucket

Replace $SourceBucket with the name of the bucket where the copy is currently stored, and $TargetBucket with the name of the bucket the copy will be stored in.

Importing copied objects from a different project

To import a copy of objects in a bucket to another bucket in a different Project, you need to implement a bucket policy.

A Bucket Policy is a resource-based policy option. In this context, Bucket Policies are used to grant the target Project access to the source bucket.

Important:

Before importing a copy, make sure all of the source bucket’s objects are in STANDARD class. If an object is in GLACIER class, it will not be copied nor imported to the target bucket.

Create a file with your Bucket Policy for the source bucket.

Note:

Bucket policies use a JSON-based access policy language. You can find more details about the JSON policy grammar on this page.

{
        "Version": "2012-10-17",
        "Id":"MyBucketPolicy",
        "Statement": [
                {
                        "Sid": "DelegateS3Access",
                        "Effect":"Allow",
                        "Principal":{
                                "SCW":"project_id:<TARGET_PROJECT_ID>"
                        },
                        "Action":[
                                "s3:ListBucket",
                                "s3:GetObject"
                        ],
                        "Resource":[
                                "<BUCKET_NAME>",
                                "<BUCKET_NAME>/*"
                        ]
                }
        ]
}

Run the following command:

$ aws s3api put-bucket-policy --bucket <SOURCE_BUCKET> --profile <SOURCE_PROJECT> --policy file://bucket-policy.json

If necessary, create the target bucket in the target project.

$ aws s3api create-bucket --bucket <TARGET_BUCKET> --profile <TARGET_PROJECT>
{
  "Location": "/<TARGET_BUCKET>"
}

Copy objects and import using the sync command.
Replace <SOURCE_BUCKET> with the name of the bucket located in the source project and <TARGET_BUCKET> with the name of the bucket in the target project.
```
$ aws s3 sync s3://<SOURCE_BUCKET> s3://<TARGET_BUCKET> --profile <TARGET_PROJECT>
```