How to use Object Storage with s3fs

Object Storage with s3fs - Overview

During this tutorial you will learn how to use s3fs as a client for Object Storage.

Requirements

Object Storage allows you to store any kind of objects (documents, images, videos, etc.). You can retrieve them anytime and from anywhere.

For instance, you can store images and they will be accessible using HTTP. You can use the control panel to manage your storage.
As our API is S3 compatible, it can be used with many tools that are created to interact with S3 compatible Object Storage.

This tutorial details how to use the Object Storage with s3fs. Pleas note that using s3fs with Scaleway, you are limited to a maximum file size of 128GB.

Important: The version of s3fs available for installation using the systems packet manager does not support files larger than 10GB. It is therefore recommended to compile a version, including the reqired corrections, from the sourcecode repository of the tool. This tutorial will guide you through the process of fixing, compiling and installing s3fs on your machine.

There are four steps to configure and use s3fs:

Retrieving your Credentials

You can access Object Storage buckets using API Key. Refer to How to generate API Key for more information about it.

Downloading s3fs

s3fs allows Linux and Mac OS X to mount an S3 bucket via FUSE. s3fs preserves the native object format for files, to be able to use them with other tools.

Installing Dependencies

Debian and Ubuntu

To install s3fs-fuse on Debian and Ubuntu based operating systems start by installing the dependencies:

apt update && apt upgrade -y
apt -y install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config

RedHat and CentOS

To install s3fs-fuse on RedHad and CentOS based operating systems start by installing the dependencies:

yum update
yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

MacOS X

On Mac OS X, install the dependencies via Homebrew:

brew cask install osxfuse 
brew install autoconf automake pkg-config gnutls libgcrypt nettle git

Installing s3fs-fuse

1 . Download the Git repository of s3fs-fuse:

git clone https://github.com/s3fs-fuse/s3fs-fuse.git

2 . Enter the s3fs-fuse directory

cd s3fs-fuse

3 . Update the MAX_MULTIPART_CNT value in the fdcache.cpp file:

  • On Linux:
    sed -i 's/MAX_MULTIPART_CNT = 10 /MAX_MULTIPART_CNT = 1 /' src/fdcache.cpp
  • On MacOS:
    sed -i '' -e 's/MAX_MULTIPART_CNT = 10 /MAX_MULTIPART_CNT = 1 /' src/fdcache.cpp

4 . Run the autogen.sh script to generate a configuration file, configure the application and compile it from the master branch:

./autogen.sh
./configure
make

5 . Then run the installation of the application using the make install command:

make install

6 . Copy the application into its final destination to complete the installation:

cp ~/s3fs-fuse/src/s3fs /usr/local/bin/s3fs

Configuring s3fs

1 . Enter your S3 credentials in a file ~/.passwd-s3fs1 and set owner-only permissions:

echo <ACCESS_KEY>:<SECRET_KEY> >  ~/.passwd-s3fs
chmod 600  ~/.passwd-s3fs

2 . Create a file system from an existing bucket:

Note:

  • Replace $SCW-BUCKET-NAME with the name of your Object Storage bucket and $FOLDER-TO-MOUNT with the local folder to mount it
  • Replace the endpoint parameter with the location of the your bucket (fr-par for Paris, nl-ams for Amsterdam or pl-waw for Warsaw)
  • Replace s3.fr-par.scw.cloud with the address of the storage cluster of your bucket. It can either be s3.nl-ams.scw.cloud (Amsterdam, The Netherlands), s3.fr-par.scw.cloud (Paris, France) or s3.pl-waw.scw.cloud (Warsaw, Poland).
s3fs $SCW-BUCKET-NAME $FOLDER-TO-MOUNT -o allow_other -o passwd_file=~/.passwd-s3fs -o use_path_request_style -o endpoint=fr-par -o parallel_count=15 -o multipart_size=128 -o nocopyapi -o url=https://s3.fr-par.scw.cloud

Important:

The flag -o multipart_size=128 sets the chunk (file-part) size for multipart uploads to 128MB. This value allows you to upload files up to a maxium file size of 128GB. This value allows you to upload files up to a maxium file size of 128GB. Lower values will give you better performances. You can set it to:

  • A minimum chunk size of 5 MB, to increase performance (Maximum file size: 5GB)
  • A maximum chunk size of 5000 MB, to increase the maxium file size (Maximum file size: 5TB)

3 . Add the following line to /etc/fstab to mount the file system on boot:

Note: Remember to replace s3.fr-par.scw.cloud with the address corresponding to your buckets’ geographical location.

s3fs#[bucket_name] /mount-point fuse _netdev,allow_other,use_path_request_style,url=https://s3.fr-par.scw.cloud/ 0 0

Using Object Storage with s3fs

The file system will appear in your OS like a local file system and you can access the files like as they are on your hard drive.

Please note, that there are some limitations when using S3 as a file system:

  • Random writes or appends to files require rewriting the entire file
  • Metadata operations such as listing directories have poor performance due to network latency
  • Eventual consistency can temporarily yield stale data
  • No atomic renames of files or directories
  • No coordination between multiple clients mounting the same bucket
  • No hard links

Discover the Cloud That Makes Sense

Get started now