Jump toSuggest an edit

Using Object Storage with s3fs

Reviewed on 18 December 2023Published on 16 July 2018
  • object-storage
  • s3fs

In this tutorial you learn how to use s3fs as a client for Scaleway Object Storage. s3fs is a FUSE-backed file interface for S3, allowing you to mount your S3 buckets on your local Linux or macOS operating system. s3fs preserves the native object format for files, so they can be used with other tools including AWS CLI.


The version of s3fs available for installation using the systems package manager does not support files larger than 10 GB. It is therefore recommended to compile a version, including the required corrections, from the s3fs source code repository. This tutorial will guide you through that process. Note that even with the source code compiled version of s3fs, there is a maximum file size of 128 GB when using s3fs with Scaleway Object Storage.

Before you start

To complete the actions presented below, you must have:

  • A Scaleway account logged into the console
  • Owner status or IAM permissions allowing you to perform actions in the intended Organization
  • A valid API key

Installing s3fs


Start by installing the dependencies of s3fs-fuse by executing the following commands, depending on your operating system:

  • On Debian and Ubuntu, from the command line:
    apt update && apt upgrade -y
    apt -y install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
  • On CentOS, from the command line:
    dnf update
    dnf install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
  • On macOS, via Homebrew:
    brew install --cask osxfuse
    brew install autoconf automake pkg-config gnutls libgcrypt nettle git

On macOS you need to add permissions to FUSE. Go to the Settings > Security & Privacy > General tab to allow the extension.


Next, download and install s3fs-fuse itself:

  1. Download the Git repository of s3fs-fuse:

    git clone
  2. Enter the s3fs-fuse directory

    cd s3fs-fuse
  3. Update the MAX_MULTIPART_CNT value in the fdcache_entity.cpp file:

    • On Linux:
    sed -i 's/MAX_MULTIPART_CNT = 10 /MAX_MULTIPART_CNT = 1 /' src/fdcache_entity.cpp
    • On macOS:
    sed -i '' -e 's/MAX_MULTIPART_CNT = 10 /MAX_MULTIPART_CNT = 1 /' src/fdcache_entity.cpp
  4. Run the script to generate a configuration file, configure the application and compile it from the master branch:

  5. Run the installation of the application using the make install command:

    make install
  6. Copy the application into its final destination to complete the installation:

    cp ~/s3fs-fuse/src/s3fs /usr/local/bin/s3fs

Configuring s3fs

  1. Execute the following commands to enter your S3 credentials (seperated by a :) in a file $HOME/.passwd-s3fs and set owner-only permissions. This presumes that you have set your API credentials as environment variables named ACCESS_KEY and SECRET_KEY:

    echo $ACCESS_KEY:$SECRET_KEY > $HOME/.passwd-s3fs
    chmod 600 $HOME/.passwd-s3fs
  2. Execute the following commands to create a file system from an existing bucket. Make the following replacements in the command text:

    • Replace $SCW-BUCKET-NAME with the name of your Object Storage bucket and $FOLDER-TO-MOUNT with the local folder to mount it in.
    • Replace the endpoint parameter with the location of your bucket (fr-par for Paris, nl-ams for Amsterdam or pl-waw for Warsaw).
    • Replace with the address of the storage cluster of your bucket. It can either be (Amsterdam, The Netherlands), (Paris, France) or (Warsaw, Poland).
    s3fs $SCW-BUCKET-NAME $FOLDER-TO-MOUNT -o allow_other -o passwd_file=$HOME/.passwd-s3fs -o use_path_request_style -o endpoint=fr-par -o parallel_count=15 -o multipart_size=128 -o nocopyapi -o url=

    The flag -o multipart_size=128 sets the chunk (file-part) size for multipart uploads to 128 MB. This value allows you to upload files up to a maximum file size of 128 GB. Lower values will give you better performances. You can set it to:

    • A minimum chunk size of 5 MB, to increase performance (Maximum file size: 5 GB)
    • A maximum chunk size of 5000 MB, to increase the maximum file size (Maximum file size: 5 TB)

    You must carry out the command as root in order for the allow_other argument to be allowed.

  3. Add the following line to /etc/fstab to mount the file system on boot. Replace with the address corresponding to your bucket’s geographical location:

    s3fs/#[bucket_name] /mount-point fuse _netdev,allow_other,use_path_request_style,url= 0 0

Using Object Storage with s3fs

The file system of the mounted bucket will appear in your OS like a local file system. This means you can access the files as if they were on your hard drive.

Note that there are some limitations when using S3 as a file system:

  • Random writes or appends to files require rewriting the entire file
  • Metadata operations such as listing directories have poor performance due to network latency
  • Eventual consistency can temporarily yield stale data
  • No atomic renames of files or directories
  • No coordination between multiple clients mounting the same bucket
  • No hard links.
Docs APIScaleway consoleDedibox consoleScaleway LearningScaleway.comPricingBlogCarreer
© 2023-2024 – Scaleway