Using Object Storage with s3fs

Reviewed on 18 December 2023 • Published on 16 July 2018

object-storage
s3fs

In this tutorial you learn how to use s3fs as a client for Scaleway Object Storage. s3fs is a FUSE-backed file interface for S3, allowing you to mount your S3 buckets on your local Linux or macOS operating system. s3fs preserves the native object format for files, so they can be used with other tools including AWS CLI.

Important

The version of s3fs available for installation using the systems package manager does not support files larger than 10 GB. It is therefore recommended to compile a version, including the required corrections, from the s3fs source code repository. This tutorial will guide you through that process. Note that even with the source code compiled version of s3fs, there is a maximum file size of 128 GB when using s3fs with Scaleway Object Storage.

Before you start

To complete the actions presented below, you must have:

A Scaleway account logged into the console
Owner status or IAM permissions allowing you to perform actions in the intended Organization
A valid API key

Installing s3fs

Dependencies

Start by installing the dependencies of s3fs-fuse by executing the following commands, depending on your operating system:

On Debian and Ubuntu, from the command line:

apt update && apt upgrade -y
apt -y install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config

On CentOS, from the command line:

dnf update
dnf install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

On macOS, via Homebrew:
```
brew install --cask osxfuse
brew install autoconf automake pkg-config gnutls libgcrypt nettle git
```

Note

On macOS you need to add permissions to FUSE. Go to the Settings > Security & Privacy > General tab to allow the extension.

s3fs-fuse

Next, download and install s3fs-fuse itself:

Download the Git repository of s3fs-fuse:

git clone https://github.com/s3fs-fuse/s3fs-fuse.git

Enter the s3fs-fuse directory

cd s3fs-fuse

Update the MAX_MULTIPART_CNT value in the fdcache_entity.cpp file:

On Linux:

sed -i 's/MAX_MULTIPART_CNT         = 10 /MAX_MULTIPART_CNT         = 1 /' src/fdcache_entity.cpp

On macOS:

sed -i '' -e 's/MAX_MULTIPART_CNT         = 10 /MAX_MULTIPART_CNT         = 1 /' src/fdcache_entity.cpp

Run the autogen.sh script to generate a configuration file, configure the application and compile it from the master branch:

./autogen.sh
./configure
make

Run the installation of the application using the make install command:

make install

Copy the application into its final destination to complete the installation:

cp ~/s3fs-fuse/src/s3fs /usr/local/bin/s3fs

Configuring s3fs

Execute the following commands to enter your S3 credentials (seperated by a :) in a file $HOME/.passwd-s3fs and set owner-only permissions. This presumes that you have set your API credentials as environment variables named ACCESS_KEY and SECRET_KEY:
```
echo $ACCESS_KEY:$SECRET_KEY > $HOME/.passwd-s3fs
chmod 600 $HOME/.passwd-s3fs
```
Execute the following commands to create a file system from an existing bucket. Make the following replacements in the command text:
- Replace $SCW-BUCKET-NAME with the name of your Object Storage bucket and $FOLDER-TO-MOUNT with the local folder to mount it in.
- Replace the endpoint parameter with the location of your bucket (fr-par for Paris, nl-ams for Amsterdam or pl-waw for Warsaw).
- Replace s3.fr-par.scw.cloud with the address of the storage cluster of your bucket. It can either be s3.nl-ams.scw.cloud (Amsterdam, The Netherlands), s3.fr-par.scw.cloud (Paris, France) or s3.pl-waw.scw.cloud (Warsaw, Poland).
```
s3fs $SCW-BUCKET-NAME $FOLDER-TO-MOUNT -o allow_other -o passwd_file=$HOME/.passwd-s3fs -o use_path_request_style -o endpoint=fr-par -o parallel_count=15 -o multipart_size=128 -o nocopyapi -o url=https://s3.fr-par.scw.cloud
```
Important
The flag -o multipart_size=128 sets the chunk (file-part) size for multipart uploads to 128 MB. This value allows you to upload files up to a maximum file size of 128 GB. Lower values will give you better performances. You can set it to:
- A minimum chunk size of 5 MB, to increase performance (Maximum file size: 5 GB)
- A maximum chunk size of 5000 MB, to increase the maximum file size (Maximum file size: 5 TB)
Note
You must carry out the command as root in order for the allow_other argument to be allowed.

Add the following line to /etc/fstab to mount the file system on boot. Replace s3.fr-par.scw.cloud with the address corresponding to your bucket’s geographical location:

s3fs/#[bucket_name] /mount-point fuse _netdev,allow_other,use_path_request_style,url=https://s3.fr-par.scw.cloud/ 0 0

Using Object Storage with s3fs

The file system of the mounted bucket will appear in your OS like a local file system. This means you can access the files as if they were on your hard drive.

Note that there are some limitations when using S3 as a file system:

Random writes or appends to files require rewriting the entire file
Metadata operations such as listing directories have poor performance due to network latency
Eventual consistency can temporarily yield stale data
No atomic renames of files or directories
No coordination between multiple clients mounting the same bucket
No hard links.