Using Scaleway Object Storage with s3fs

• object-storage
• s3fs

In this tutorial you learn how to use s3fs as a client for Scaleway Object Storage. s3fs is a FUSE-backed file interface for S3, allowing you to mount Object Storage buckets on your local Linux or macOS system. Files are preserved in their native object format, enabling compatibility with tools like AWS CLI.

Before you startLink to this anchor

To complete the actions presented below, you must have:

• A Scaleway account logged into the console
• Owner status or IAM permissions allowing you to perform actions in the intended Organization
• A valid API key

Installing s3fsLink to this anchor

Option 1: Install via package managerLink to this anchor

Debian/Ubuntu

sudo apt update && sudo apt upgrade -y
sudo apt install -y s3fs

CentOS/RHEL

sudo dnf update -y
sudo dnf install -y epel-release
sudo dnf install -y s3fs-fuse

macOS (via Homebrew)

brew install --cask macfuse
brew install gromgit/fuse/s3fs-mac

Option 2: Compile from sourceLink to this anchor

If the version provided by the package manager does not meet your requirements or you require special features, compile the tool from source.

Install dependencies

• Debian/Ubuntu:

  sudo apt install -y automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
  

• CentOS/RHEL:

  sudo dnf install -y automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
  

• macOS:

  brew install autoconf automake pkg-config gnutls libgcrypt nettle git
  brew install --cask macfuse
  

Build and install

Run the following commands to download the latest version of s3fs and build and install it on your machine:

git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make
sudo make install

Configuring s3fsLink to this anchor

1. Create a credentials file:

   echo ACCESS_KEY:SECRET_KEY > $HOME/.passwd-s3fs
   chmod 600 $HOME/.passwd-s3fs
   

   Note

   Replace ACCESS_KEY and SECRET_KEY with your Scaleway credentials.

2. Create a mount point directory:

   mkdir -p /path/to/mountpoint
   

3. Mount the bucket:

   s3fs BUCKET_NAME /path/to/mountpoint \
   -o allow_other \
   -o passwd_file=$HOME/.passwd-s3fs \
   -o use_path_request_style \
   -o endpoint=fr-par \
   -o parallel_count=15 \
   -o multipart_size=128 \
   -o nocopyapi \
   -o url=https://s3.fr-par.scw.cloud
   

   Replace:

   • BUCKET_NAME: Your bucket name.
   • /path/to/mountpoint: Path to mount the bucket.
   • fr-par: Replace with your bucket’s region (nl-ams, pl-waw).
   • s3.fr-par.scw.cloud: Replace with the endpoint URL matching the region of your bucket.

4. Configure automount on boot by adding the following line to /etc/fstab:

   s3fs#BUCKET_NAME /path/to/mountpoint fuse _netdev,allow_other,use_path_request_style,url=https://s3.fr-par.scw.cloud 0 0
   

Once mounted, the bucket behaves like a local filesystem. You can copy, move, and manage files directly.

Note that there are some limitations when using Object Storage as a file system:

• Random writes or appends to files require rewriting the entire file
• Metadata operations such as listing directories have poor performance due to network latency
• Eventual consistency can temporarily yield stale data
• No atomic renames of files or directories
• No coordination between multiple clients mounting the same bucket
• No hard links.

Troubleshooting TipsLink to this anchor

• Permission Issues: Ensure /etc/fuse.conf contains user_allow_other.
• Mount Failures: Check logs with dmesg or /var/log/syslog.
• Performance: Adjust multipart_size (e.g., 5 MB for faster uploads, 5000 MB for larger files).
• Reconnection on Network Loss: Remount manually or automate reconnections with a script.