Jump toUpdate content

Using Object Storage with s3fs

Reviewed on 01 December 2021Published on 16 July 2018
  • object-storage
  • s3fs

In this tutorial you will learn how to use s3fs as a client for Scaleway Object Storage. s3fs is a FUSE-backed file interface for S3, allowing you to mount your S3 buckets on your local Linux or macOS operating system. s3fs preserves the native object format for files, so they can be used them with other tools including AWS CLI.

Important:

The version of s3fs available for installation using the systems package manager does not support files larger than 10GB. It is therefore recommended to compile a version, including the reqired corrections, from the s3fs source code repository. This tutorial will guide you through that process. Please note that even with the source code compiled version of s3fs, there is a maximum file size of 128GB when using s3fs with Scaleway Object Storage.

Requirements:

Installing s3fs

Dependencies

Start by installing the dependencies of s3fs-fuse by executing the following commands, depending on your operating system:

  • On Debian and Ubuntu, from the command line:

    apt update && apt upgrade -y
    apt -y install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
  • On RedHat and CentOS, from the command line:

    yum update
    yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
  • On macOS, via Homebrew:

    brew install --cask osxfuse
    brew install autoconf automake pkg-config gnutls libgcrypt nettle git
    Note:

    On macOS Big Sur you need to add permissions to FUSE. Go to the Settings > Security & Privacy > General tab to allow the extension.

s3fs-fuse

Next, download and install s3fs-fuse itself:

  1. Download the Git repository of s3fs-fuse:

    git clone https://github.com/s3fs-fuse/s3fs-fuse.git
  2. Enter the s3fs-fuse directory

    cd s3fs-fuse
  3. Update the MAX_MULTIPART_CNT value in the fdcache.cpp file:

    • On Linux:
    sed -i 's/MAX_MULTIPART_CNT = 10 /MAX_MULTIPART_CNT = 1 /' src/fdcache.cpp
    • On macOS:
    sed -i '' -e 's/MAX_MULTIPART_CNT = 10 /MAX_MULTIPART_CNT = 1 /' src/fdcache.cpp
  4. Run the autogen.sh script to generate a configuration file, configure the application and compile it from the master branch:

    ./autogen.sh
    ./configure
    make
  5. Run the installation of the application using the make install command:

    make install
  6. Copy the application into its final destination to complete the installation:

    cp ~/s3fs-fuse/src/s3fs /usr/local/bin/s3fs

Configuring s3fs

  1. Execute the following commands to enter your S3 credentials (seperated by a :) in a file $HOME/.passwd-s3fs and set owner-only permissions. This presumes that you have set your API credentials as environment variables named ACCESS_KEY and SECRET_KEY:

    echo $ACCESS_KEY:$SECRET_KEY > $HOME/.passwd-s3fs
    chmod 600 $HOME/.passwd-s3fs
  2. Execute the following commands to create a file system from an existing bucket. Make the following replacements in the command text:

    • Replace $SCW-BUCKET-NAME with the name of your Object Storage bucket and $FOLDER-TO-MOUNT with the local folder to mount it in.
    • Replace the endpoint parameter with the location of the your bucket (fr-par for Paris, nl-ams for Amsterdam or pl-waw for Warsaw).
    • Replace s3.fr-par.scw.cloud with the address of the storage cluster of your bucket. It can either be s3.nl-ams.scw.cloud (Amsterdam, The Netherlands), s3.fr-par.scw.cloud (Paris, France) or s3.pl-waw.scw.cloud (Warsaw, Poland).
    s3fs $SCW-BUCKET-NAME $FOLDER-TO-MOUNT -o allow_other -o passwd_file=$HOME/.passwd-s3fs -o use_path_request_style -o endpoint=fr-par -o parallel_count=15 -o multipart_size=128 -o nocopyapi -o url=https://s3.fr-par.scw.cloud
    Important:

    The flag -o multipart_size=128 sets the chunk (file-part) size for multipart uploads to 128MB. This value allows you to upload files up to a maximum file size of 128GB. Lower values will give you better performances. You can set it to:

    • A minimum chunk size of 5 MB, to increase performance (Maximum file size: 5GB)
    • A maximum chunk size of 5000 MB, to increase the maximum file size (Maximum file size: 5TB)
    Note:

    You must carry out the command as root in order for the allow_other argument to be allowed.

  3. Add the following line to /etc/fstab to mount the file system on boot. Replace s3.fr-par.scw.cloud with the address corresponding to your bucket’s geographical location:

    s3fs#[bucket_name] /mount-point fuse _netdev,allow_other,use_path_request_style,url=https://s3.fr-par.scw.cloud/ 0 0

Using Object Storage with s3fs

The file system of the mounted bucket will appear in your OS like a local file system. This means you can access the files as if they were on your hard drive.

Please note, that there are some limitations when using S3 as a file system:

  • Random writes or appends to files require rewriting the entire file
  • Metadata operations such as listing directories have poor performance due to network latency
  • Eventual consistency can temporarily yield stale data
  • No atomic renames of files or directories
  • No coordination between multiple clients mounting the same bucket
  • No hard links