How to Import a Copy of S3 Objects from one Bucket to Another

Import Copy Overview

Object Storage allows you to import a copy of your objects in a bucket to another bucket using the AWS-CLI.

The sync command is used to synchronize directories and S3 prefixes.

This command will:

  • Recurringly copy new and updated files from the source directory to the target one
  • Use the CopyObject APIs to list objects in the source and target buckets and identify which ones they have in common or not
  • Compare both to determine disparities in LastModified dates between the same objects

Note: When applied on a versioned bucket, the sync command copies only the most recent version of the objects.

Requirements

You have:

Importing Copied Objects from the Same Project

To import copied objects to a bucket in the same Project, use the aws s3 sync command.

Important: Before importing a copy, make sure all of the source bucket’s objects are in STANDARD class. If an object is in GLACIER class, the copy import will fail.

aws s3 sync s3://$SourceBucket s3://$TargetBucket

Replace $SourceBucket with the name of the bucket where the copy is currently stored in and $TargetBucket with the name of the bucket the copy will be stored in.

Importing Copied Objects from a Different Project

To import a copy of objects in a bucket to another bucket in a different Project, you need to implement a Bucket Policy.

A Bucket Policy is a resource-based policy option. In this context, Bucket Policies are used to grant the target Project access to the source bucket.

Note: Bucket policies use a JSON-based access policy language. You can find more details about the JSON policy grammar on this page.

PUT Bucket Policy

This operation applies an S3 bucket policy to an S3 bucket.

Sample API Request:

PUT /myBucket?policy HTTP/1.1
{
    "Version": "2012-10-17",
    "Id": "MyBucketPolicy",
    "Statement": [
        {
            "Sid": "DelegateAccess",
            "Effect": "Allow",
            "Principal":{
                "SCW": "project_id:<PROJECT_ID>"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "<BUCKET_NAME>",
                "<BUCKET_NAME>/*"
            ]
        }
    ]
}

AWS-CLI Command:

Create the file bucket-policy.json with the following content:

{
    "Version": "2012-10-17",
    "Id": "Mybucketpolicy",
    "Statement": [
        {
            "Sid": "DelegateAccess",
            "Effect": "Allow",
            "Principal": {
                "SCW": "project_id:<PROJECT_ID>"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "<BUCKET_NAME>",
                "<BUCKET_NAME>/*"
            ]
        }
    ]
}

Then run the following command:

$ aws s3api put-bucket-policy --bucket <BUCKET_NAME> --policy file://bucket-policy.json

GET Bucket Policy

This operation returns the policy of a specified bucket.

Sample API Request:

GET /myBucket?policy HTTP/1.1

Sample API Output:

{
    "Policy": "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Action\": [\"s3:GetObject\"], \"Principal\": {\"SCW\": [\"<PROJECT_ID>\"]}, \"Resource\": [\"myBucket/*\"], \"Effect\": \"Allow\", \"Sid\": \"DelegateGetObject\"}]}"
}

AWS-CLI Command:

$ aws s3api get-bucket-policy --bucket myBucket
{
    "Policy": "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Action\": [\"s3:GetObject\"], \"Principal\": {\"SCW\": [\"<PROJECT_ID>\"]}, \"Resource\": [\"myBucket/*\"], \"Effect\": \"Allow\", \"Sid\": \"DelegateGetObject\"}]}"
}

DELETE Bucket Policy

This operation deletes the Bucket Policy of a specified bucket.

If the operation is successful no output will be returned.

Sample API Request:

DELETE /MyBucket?policy HTTP/1.1

AWS-CLI Command:

$ aws s3api delete-bucket-policy --bucket myBucket -> code block

Importing the Copied Objects

To import a copy of objects in a bucket to another bucket in a different Project, you must grant the target Project access to the source bucket.

Important: Before importing a copy, make sure all of the source bucket’s objects are in STANDARD class. If an object is in GLACIER class, it will not be copied nor imported to the target bucket.

1 . Set a Bucket Policy for your source bucket.

Create the file bucket-policy.json with the following content:

{
    "Version": "2012-10-17",
    "Id":"MyBucketPolicy",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect":"Allow",
            "Principal":{
                "SCW":"project_id:<TARGET_PROJECT_ID>"
            },
            "Action":[
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource":[
                "<BUCKET_NAME>",
                "<BUCKET_NAME>/*"
            ]
        }
    ]
}

Then run the following command:

$ aws s3api put-bucket-policy --bucket <SOURCE_BUCKET> --profile <SOURCE_PROJECT> --policy file://bucket-policy.json

2 . If necessary, create the target bucket in the target project.

$ aws s3api create-bucket --bucket <TARGET_BUCKET> --profile <TARGET_PROJECT>
{
    "Location": "/<TARGET_BUCKET>"
}

3 . Copy objects and import using the sync command.

Replace <SOURCE_BUCKET> with the name of the bucket located in the source Project and <TARGET_BUCKET> with the name of the bucket in the target Project.

$ aws s3api create-bucket --bucket <TARGET_BUCKET> --profile <TARGET_PROJECT>

Discover the Cloud That Makes Sense