Using Object Storage with s3fs
In this tutorial you learn how to use s3fs as a client for Scaleway Object Storage.
s3fs is a FUSE-backed file interface for S3, allowing you to mount your S3 buckets on your local Linux or macOS operating system.
s3fs preserves the native object format for files, so they can be used with other tools including AWS CLI.
You may need certain IAM permissions to carry out some actions described on this page. This means:
The version of
s3fs available for installation using the systems package manager does not support files larger than 10 GB. It is therefore recommended to compile a version, including the required corrections, from the s3fs source code repository. This tutorial will guide you through that process. Note that even with the source code compiled version of s3fs, there is a maximum file size of 128 GB when using s3fs with Scaleway Object Storage.
Start by installing the dependencies of
s3fs-fuse by executing the following commands, depending on your operating system:
- On Debian and Ubuntu, from the command line:
apt update && apt upgrade -yapt -y install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
- On RedHat and CentOS, from the command line:
yum updateyum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
- On macOS, via Homebrew:
brew install --cask osxfusebrew install autoconf automake pkg-config gnutls libgcrypt nettle git
On macOS you need to add permissions to FUSE. Go to the
Settings > Security & Privacy > General tab to allow the extension.
Next, download and install
Download the Git repository of
s3fs-fuse:git clone https://github.com/s3fs-fuse/s3fs-fuse.git
Enter the s3fs-fuse directorycd s3fs-fuse
MAX_MULTIPART_CNTvalue in the
sed -i 's/MAX_MULTIPART_CNT = 10 /MAX_MULTIPART_CNT = 1 /' src/fdcache_entity.cpp
- On Linux:
sed -i '' -e 's/MAX_MULTIPART_CNT = 10 /MAX_MULTIPART_CNT = 1 /' src/fdcache_entity.cpp
- On macOS:
autogen.shscript to generate a configuration file, configure the application and compile it from the master branch:./autogen.sh./configuremake
Run the installation of the application using the
make installcommand:make install
Copy the application into its final destination to complete the installation:cp ~/s3fs-fuse/src/s3fs /usr/local/bin/s3fs
Execute the following commands to enter your S3 credentials (seperated by a
:) in a file
$HOME/.passwd-s3fsand set owner-only permissions. This presumes that you have set your API credentials as environment variables named
SECRET_KEY:echo $ACCESS_KEY:$SECRET_KEY > $HOME/.passwd-s3fschmod 600 $HOME/.passwd-s3fs
Execute the following commands to create a file system from an existing bucket. Make the following replacements in the command text:
s3fs $SCW-BUCKET-NAME $FOLDER-TO-MOUNT -o allow_other -o passwd_file=$HOME/.passwd-s3fs -o use_path_request_style -o endpoint=fr-par -o parallel_count=15 -o multipart_size=128 -o nocopyapi -o url=https://s3.fr-par.scw.cloudImportant:
$SCW-BUCKET-NAMEwith the name of your Object Storage bucket and
$FOLDER-TO-MOUNTwith the local folder to mount it in.
- Replace the
endpointparameter with the location of your bucket (
nl-amsfor Amsterdam or
s3.fr-par.scw.cloudwith the address of the storage cluster of your bucket. It can either be
s3.nl-ams.scw.cloud(Amsterdam, The Netherlands),
s3.fr-par.scw.cloud(Paris, France) or
-o multipart_size=128sets the chunk (file-part) size for multipart uploads to 128 MB. This value allows you to upload files up to a maximum file size of 128 GB. Lower values will give you better performances. You can set it to:
- A minimum chunk size of 5 MB, to increase performance (Maximum file size: 5 GB)
- A maximum chunk size of 5000 MB, to increase the maximum file size (Maximum file size: 5 TB)
You must carry out the command as root in order for the
allow_otherargument to be allowed.
Add the following line to
/etc/fstabto mount the file system on boot. Replace
s3.fr-par.scw.cloudwith the address corresponding to your bucket’s geographical location:s3fs/#[bucket_name] /mount-point fuse _netdev,allow_other,use_path_request_style,url=https://s3.fr-par.scw.cloud/ 0 0
The file system of the mounted bucket will appear in your OS like a local file system. This means you can access the files as if they were on your hard drive.
Note that there are some limitations when using S3 as a file system:
- Random writes or appends to files require rewriting the entire file
- Metadata operations such as listing directories have poor performance due to network latency
- Eventual consistency can temporarily yield stale data
- No atomic renames of files or directories
- No coordination between multiple clients mounting the same bucket
- No hard links