Jump toUpdate content

Using Object Storage with the AWS-CLI

Reviewed on 14 June 2021Published on 16 July 2018

The AWS-CLI is an open-source tool built on top of the AWS SDK for Python (Boto) that provides commands for interacting with AWS services. With minimal configuration, you can start using all of the functionalities provided by the AWS Management.

Requirements:
Note:

This page uses AWS-CLI v1. If you want to follow the installation procedure for v2, see the AWS-CLI documentation page.

How to install the AWS-CLI

The AWS-CLI is an open-source tool built on top of the AWS SDK for Python (Boto) that provides commands for interacting with AWS services. With minimal configuration, you can start using all of the functionality provided by the AWS Management.

To interact with AWS, aws-cli and awscli-plugin-endpoint need to be installed. The awscli-plugin-endpoint is a great plugin to help people more easily access third-party S3 providers.

  1. Install aws-cli and awscli-plugin using pip.

    pip3 install awscli
    pip3 install awscli-plugin-endpoint
  2. Create the file ~/.aws/config by running the following command:

    aws configure set plugins.endpoint awscli_plugin_endpoint
  3. Open ~/.aws/config in a text editor and edit it as follows:

    [plugins]
    endpoint = awscli_plugin_endpoint

    [default]
    region = nl-ams
    s3 =
    endpoint_url = https://s3.nl-ams.scw.cloud
    signature_version = s3v4
    max_concurrent_requests = 100
    max_queue_size = 1000
    multipart_threshold = 50MB
    # Edit the multipart_chunksize value according to the file sizes that you want to upload. The present configuration allows to upload files up to 10 GB (100 requests * 10MB). For example setting it to 5GB allows you to upload files up to 5TB.
    multipart_chunksize = 10MB
    s3api =
    endpoint_url = https://s3.nl-ams.scw.cloud

    You can also configure additional profiles by adding new blocks under [default]. For example, you can add a second profile, [profile two], to set a different region and endpoint from your default one:

    [plugins]
    endpoint = awscli_plugin_endpoint

    [default]
    region = nl-ams
    s3 =
    endpoint_url = https://s3.nl-ams.scw.cloud
    s3api =
    endpoint_url = https://s3.nl-ams.scw.cloud

    [profile two]
    region = fr-par
    s3 =
    endpoint_url = https://s3.fr-par.scw.cloud
    s3api =
    endpoint_url = https://s3.fr-par.scw.cloud
    Important:

    If you are using the AWS-CLI v2, you must include the path to the plugin in your configuration file. Add cli_legacy_plugin_path = <path-to-plugin> to the [plugins] section, replacing <path-to-plugin> with the corresponding path.

    Note:

    Set the endpoint_url and region corresponding to the geographical region of your bucket. It can either be fr-par (Paris, France), nl-ams (Amsterdam, The Netherlands) or pl-waw (Warsaw, Poland).

  4. Generate a credentials file using the command:

    aws configure
  5. Open the ~/.aws/credentials file and configure your API key as follows:

    [default]
    aws_access_key_id=<ACCESS_KEY>
    aws_secret_access_key=<SECRET_KEY>

    If you have other profiles, add a block to indicate their credentials.

    [two]
    aws_access_key_id=<ACCESS_KEY>
    aws_secret_access_key=<SECRET_KEY>
  6. Test your cluster.

    aws s3 ls

    Use the --profile option if you want to test it using a different profile. Remember to replace two with the name of your profile.

    aws --profile two s3 ls
See Also