Jump toUpdate content

Importing a copy of S3 Objects from one bucket to another

Reviewed on 28 December 2021Published on 27 May 2021

Object Storage allows you to import a copy of your objects in a bucket to another bucket using the AWS-CLI.

The sync command is used to synchronize directories and S3 prefixes.

This command will:

  • Recurringly copy new and updated files from the source directory to the target one
  • Use the CopyObject APIs to list objects in the source and target buckets and identify which ones they have in common
  • Compare both to determine disparities in LastModified dates between the same objects
Note:

When applied on a versioned bucket, the sync command copies only the most recent version of the objects.

Requirements:

Importing copied objects from the same project

To import copied objects to a bucket in the same Project, use the aws s3 sync command.

Important:

Before importing a copy, make sure all of the source bucket’s objects are in one of the two classes: STANDARD or ONEZONE_IA (only available in FR-PAR). If an object is in GLACIER class, the copy import will fail. To learn how to edit storage classes, refer to this documentation page.

aws s3 sync s3://$SourceBucket s3://$TargetBucket

Replace $SourceBucket with the name of the bucket where the copy is currently stored, and $TargetBucket with the name of the bucket the copy will be stored in.

Importing copied objects from a different project

To import a copy of objects in a bucket to another bucket in a different Project, you need to implement a bucket policy.

A bucket policy is a resource-based policy option. In this context, bucket policies are used to grant the target Project access to the source bucket.

Important:

Before importing a copy, make sure all of the source bucket’s objects are in one of the two classes: STANDARD or ONEZONE_IA (only available in FR-PAR). If an object is in GLACIER class, the copy import will fail. To learn how to edit storage classes, refer to this documentation page.

  1. Create a file with your bucket policy for the source bucket.

    Note:

    Bucket policies use a JSON-based access policy language. You can find more details about the JSON policy grammar on this page.

    {
    "Version": "2012-10-17",
    "Id":"MyBucketPolicy",
    "Statement": [
    {
    "Sid": "DelegateS3Access",
    "Effect":"Allow",
    "Principal":{
    "SCW":"project_id:<TARGET_PROJECT_ID>"
    },
    "Action":[
    "s3:ListBucket",
    "s3:GetObject"
    ],
    "Resource":[
    "<BUCKET_NAME>",
    "<BUCKET_NAME>/*"
    ]
    }
    ]
    }

  2. Run the following command:

    $ aws s3api put-bucket-policy --bucket <SOURCE_BUCKET> --profile <SOURCE_PROJECT> --policy file://bucket-policy.json
  3. If necessary, create the target bucket in the target project.

    $ aws s3api create-bucket --bucket <TARGET_BUCKET> --profile <TARGET_PROJECT>
    {
    "Location": "/<TARGET_BUCKET>"
    }
  4. Copy objects and import using the sync command.

  5. Replace <SOURCE_BUCKET> with the name of the bucket located in the source project and <TARGET_BUCKET> with the name of the bucket in the target project.

    $ aws s3 sync s3://<SOURCE_BUCKET> s3://<TARGET_BUCKET> --profile <TARGET_PROJECT>