How to use Object Storage with s3fs

Using Object Storage with s3fs

During this tutorial you will learn how to use s3fs as a client for Object Storage.

Requirements

• You have an account and are logged into console.scaleway.com
• You have generated your API Token

Object Storage allows you to store any kind of objects (documents, images, videos, etc.). You can retrieve them anytime and from anywhere.

For instance, you can store images and they will be accessible using HTTP. You can use the control panel to manage your storage. As the our API is S3 compatible, a lot of tools exist to interact with the object storage.

This tutorial details how to use the Object Storage with s3fs.

There are four steps to configure and use s3fs:

• Retrieving your Credentials
• Downloading s3fs
• Configuring s3fs
• Using Object Storage with s3fs

Retrieving your Credentials

To retrieve your credentials, refer to S3 credentials.

Downloading s3fs

s3fs allows Linux and Mac OS X to mount an S3 bucket via FUSE. s3fs preserves the native object format for files, to be able to use them with other tools.

Recent operating systems provide pre-built packages, that you can install directly from their packet manager:

On Debian 9 and Ubuntu 16.04 or newer, install the application via apt:

sudo apt-get install s3fs

On SUSE 12 or newer and openSUSE 42.1 or newer install it via zippy:

sudo zypper in s3fs

On Mac OS X, install the tool via Homebrew:

brew cask install osxfuse
brew install s3fs

If your system does not provide a pre-build binary, it is possible to build the application from its sources. On Linux, ensure you have all the dependencies:

On Ubuntu 14.04:

sudo apt-get install automake autotools-dev fuse g++ git libcurl4-openssl-dev libfuse-dev libssl-dev libxml2-dev make pkg-config

On CentOS 7:

sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

Then compile the application from its master-branch via the following commands:

git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make
sudo make install

Configuring s3fs

1 . Enter your S3 credentials in a file ~/.passwd-s3fs1 and set owner-only permissions:

echo <ACCESS_KEY>:<SECRET_KEY> >  ~/.passwd-s3fs
chmod 600  ~/.passwd-s3fs

2 . Create a file system from an existing bucket:

s3fs [bucket_name] [root_directory] -o passwd_file=${HOME}/.passwd-s3fs,url=https://s3.nl-ams.scw.cloud

Note: Replace https://s3.nl-ams.scw.cloud with the address of the storage cluster of your bucket. It can either be s3.nl-ams.scw.cloud (Amsterdam, The Netherlands) or s3.fr-par.scw.cloud (Paris, France).

3 . Add the following line to /etc/fstab to mount the file system on boot:

s3fs#[bucket_name] /mount-point fuse _netdev,allow_other,use_path_request_style,url=https://s3.nl-ams.scw.cloud/ 0 0

Using Object Storage with s3fs

The file system will appear in your OS like a local file system and you can access the files like as they are on your hard drive.

Please note, that there are some limitations when using S3 as a file system:

• Random writes or appends to files require rewriting the entire file
• Metadata operations such as listing directories have poor performance due to network latency
• Eventual consistency can temporarily yield stale data
• No atomic renames of files or directories
• No coordination between multiple clients mounting the same bucket
• No hard links