Jump toUpdate content
Importing a copy of S3 Objects from one bucket to another
Object Storage allows you to import a copy of your objects in a bucket to another bucket using the AWS-CLI.
The sync
command is used to synchronize directories and S3 prefixes.
This command will:
- Recurringly copy new and updated files from the source directory to the target one
- Use the
CopyObject
APIs to list objects in the source and target buckets and identify which ones they have in common - Compare both to determine disparities in
LastModified
dates between the same objects
If you have activated IAM, you may need certain IAM permissions to carry out some actions described on this page. This means:
- you are the Owner of the Scaleway Organization in which the actions will be carried out, or
- you are an IAM user of the Organization, with a policy granting you the necessary permission sets
When applied on a versioned bucket, the sync command copies only the most recent version of the objects.
- You have an account and are logged into the Scaleway console
- You have configured your API keys
- You have created an Object Storage bucket
Importing copied objects from the same project
To import copied objects to a bucket in the same Project, use the aws s3 sync
command.
Before importing a copy, make sure all of the source bucket’s objects are in one of the two classes: STANDARD
or ONEZONE_IA
(only available in FR-PAR). If an object is in GLACIER
class, the copy import will fail. To learn how to edit storage classes, refer to this documentation page.
aws s3 sync s3://$SourceBucket s3://$TargetBucket
Replace $SourceBucket
with the name of the bucket where the copy is currently stored, and $TargetBucket
with the name of the bucket the copy will be stored in.
Importing copied objects from a different project
To import a copy of objects in a bucket to another bucket in a different Project, you need to implement a bucket policy.
A bucket policy is a resource-based policy option. In this context, bucket policies are used to grant the target Project access to the source bucket.
Before importing a copy, make sure all of the source bucket’s objects are in one of the two classes: STANDARD
or ONEZONE_IA
(only available in FR-PAR). If an object is in GLACIER
class, the copy import will fail. To learn how to edit storage classes, refer to this documentation page.
-
Create a file with your bucket policy for the source bucket.
Note:Bucket policies use a JSON-based access policy language. You can find more details about the JSON policy grammar on this page.
{
"Version": "2012-10-17",
"Id":"MyBucketPolicy",
"Statement": [
{
"Sid": "DelegateS3Access",
"Effect":"Allow",
"Principal":{
"SCW":"project_id:<TARGET_PROJECT_ID>"
},
"Action":[
"s3:ListBucket",
"s3:GetObject"
],
"Resource":[
"<BUCKET_NAME>",
"<BUCKET_NAME>/*"
]
}
]
} -
Run the following command:
$ aws s3api put-bucket-policy --bucket <SOURCE_BUCKET> --profile <SOURCE_PROJECT> --policy file://bucket-policy.json
-
If necessary, create the target bucket in the target project.
$ aws s3api create-bucket --bucket <TARGET_BUCKET> --profile <TARGET_PROJECT>
{
"Location": "/<TARGET_BUCKET>"
} -
Copy objects and import using the
sync
command. -
Replace
<SOURCE_BUCKET>
with the name of the bucket located in the source project and<TARGET_BUCKET>
with the name of the bucket in the target project.$ aws s3 sync s3://<SOURCE_BUCKET> s3://<TARGET_BUCKET> --profile <TARGET_PROJECT>