Jump toUpdate content
Importing a copy of S3 Objects from one bucket to another
Object Storage allows you to import a copy of your objects in a bucket to another bucket using the AWS-CLI.
The sync
command is used to synchronize directories and S3 prefixes.
This command will:
- Recurringly copy new and updated files from the source directory to the target one
- Use the
CopyObject
APIs to list objects in the source and target buckets and identify which ones they have in common - Compare both to determine disparities in
LastModified
dates between the same objects
When applied on a versioned bucket, the sync command copies only the most recent version of the objects.
- You have an account and are logged into the Scaleway console
- You have configured your API keys
- You have created an Object Storage bucket
Importing copied objects from the same project
To import copied objects to a bucket in the same Project, use the aws s3 sync
command.
Before importing a copy, make sure all of the source bucket’s objects are in one of the two classes: STANDARD
or ONEZONE_IA
(only available in FR-PAR). If an object is in GLACIER
class, the copy import will fail. To learn how to edit storage classes, refer to this documentation page.
aws s3 sync s3://$SourceBucket s3://$TargetBucket
Replace $SourceBucket
with the name of the bucket where the copy is currently stored, and $TargetBucket
with the name of the bucket the copy will be stored in.
Importing copied objects from a different project
To import a copy of objects in a bucket to another bucket in a different Project, you need to implement a bucket policy.
A bucket policy is a resource-based policy option. In this context, bucket policies are used to grant the target Project access to the source bucket.
Before importing a copy, make sure all of the source bucket’s objects are in one of the two classes: STANDARD
or ONEZONE_IA
(only available in FR-PAR). If an object is in GLACIER
class, the copy import will fail. To learn how to edit storage classes, refer to this documentation page.
Create a file with your bucket policy for the source bucket.
Note:Bucket policies use a JSON-based access policy language. You can find more details about the JSON policy grammar on this page.
{
"Version": "2012-10-17",
"Id":"MyBucketPolicy",
"Statement": [
{
"Sid": "DelegateS3Access",
"Effect":"Allow",
"Principal":{
"SCW":"project_id:<TARGET_PROJECT_ID>"
},
"Action":[
"s3:ListBucket",
"s3:GetObject"
],
"Resource":[
"<BUCKET_NAME>",
"<BUCKET_NAME>/*"
]
}
]
}Run the following command:
$ aws s3api put-bucket-policy --bucket <SOURCE_BUCKET> --profile <SOURCE_PROJECT> --policy file://bucket-policy.json
If necessary, create the target bucket in the target project.
$ aws s3api create-bucket --bucket <TARGET_BUCKET> --profile <TARGET_PROJECT>
{
"Location": "/<TARGET_BUCKET>"
}Copy objects and import using the
sync
command.Replace
<SOURCE_BUCKET>
with the name of the bucket located in the source project and<TARGET_BUCKET>
with the name of the bucket in the target project.$ aws s3 sync s3://<SOURCE_BUCKET> s3://<TARGET_BUCKET> --profile <TARGET_PROJECT>