Since many years now, people are using File Transfer Protocol (FTP) to back up their data on specific or internal storage servers. Most of current software are offering an FTP backup option to avoid data loss, and store your backup elsewhere than on your production server. We all know that: backup is more than criticals.

In 2006, AWS created a Simple Storage Service called S3. That's a "basic" storage service, which is built on a non-hierarchic file system. Technically it means that you can't create folders, subfolders, etc.

This protocol was made public, and most known cloud providers are offering an S3 compatible service. Wasabi, Digital Ocean, Alibaba Cloud, Oracle Cloud, Scaleway, and more other are offering a S3 compatible service. For your information you can easily create your own S3 compatible server, using Minio, but we will see that on another tutorial.

Today we will see how to deploy an FTP/SFTP/SSH/DAV server backed by an S3 bucket on any cloud provider. For that, we will use a software called SFTPGo. Let's learn how to install SFTPGo and configure it.

Warnings:
-For AWS S3 users, there is already a service offered by AWS called "'AWS Transfer for SFTP", which will allow you to get a managed SFTP server for your S3 bucket, and it's free of charges.
-All S3 providers are charging fees for outgoing data transfer (outgoing bandwidth), but some (as Scaleway), are not billing you bandwidth between your server, and your S3 bucket. Check that before buying an instance/VPS elsewhere than on your S3 provider.
-Technically, you will be "limited" to your instance/VPS "bandwidth" maximum capacity. Providers S3 endpoints, can handle gigabytes, but if your instance/VPS bandwidth is 100Mbps, it will automatically bottleneck.

Prerequisite:
-have a compute cloud instance (VPS, dedicated, instance, etc) and root permissions
-have a created bucket on any supported provider

Step 1: Login, and install SFTPGo on your instance

Let's connect to your instance/VPS as usual, with the following command:

ssh root@YourInstanceIP

Once it's done, as usual, please update and upgrade your packages installed.

apt-get update && apt-get upgrade -y
# The -y flag, will automatically accept and confirm both tasks

Now, let's move to a specific folder to download SFTPGo and wget the latest version available.
As a reminder, the latest versions are available here: https://github.com/drakkan/sftpgo/releases

cd /usr/local/sbin
# First, move to this directory
wget https://github.com/drakkan/sftpgo/releases/download/v2.0.2/sftpgo_2.0.2-1_amd64.deb
# Download the latest version available w/ your CPU architecture on the .deb format

Once you've downloaded the SFTPGo deb file, install it with dpkg:

dpkg -i sftpgo_2.0.2-1_amd64.deb

To ensure that your SFTPGo service is correctly installed, and running, perform the following command:

systemctl status sftpgo

You may have an output like this one:

If you can see "active (running)", it means that you've successfully installed SFTPGo!

Step 2: Configure SFTPGo and specific ports service + SSL

By default, SFTPGo is using many ports, for different services (WebUI, SFTP, FTP, etc). First let's check which ports, were used by default.

netstat -tulpn | grep sftpgo
# If you don't already have net-tools installed, perform apt install net-tools

Then, you will see all ports used by SFTPGo, like below.

Now, we know that SFTP is using these ports, but we want to modify to use custom ports, as by default 8080 ports may be already used.

As we will modify SFTPGo, we need to turn off, and kill all SFTPGo active services.

killall sftpgo
# It can also be done with killall sftpgo.service

Let's move on the SFTPGo directory, and list all files that we will edit:

cd /etc/sftpgo && ls

Then, you will get the following output, with all files used by SFTPGo.

On our case, we will edit only the sftpgo.json file.
So, lets open-it with your favorite text editor, and have a look at the detailed configuration.

nano sftgo.json

Between SFTPGo versions, this file may got new sections, and more, but we will focus on these ones:

-The SFTPD section: It will allow you to enable, disable, and define used port.

  "sftpd": {
    "bindings": [
      {
        "port": 2022,
        "address": "",
        "apply_proxy_config": true
      }
    ],
    "max_auth_tries": 0,
    "banner": "",
    "host_keys": [],
    "kex_algorithms": [],
    "ciphers": [],
    "macs": [],
    "trusted_user_ca_keys": [],
    "login_banner_file": "",
    "enabled_ssh_commands": [
      "md5sum",
      "sha1sum",
      "cd",
      "pwd",
      "scp"
    ],
    "keyboard_interactive_auth_hook": "",
    "password_authentication": true
  },

Here, I want to turn off SFTP service, so I just modify this line like below:

        "port": 0,

The 0 port, will define SFTP service as "inactive". And we will be able to check that later via the WebUI.

-The FTPD section: It will allow you to enable, disable, and define used port.

  "ftpd": {
    "bindings": [
      {
        "port": 0,
        "address": "",
        "apply_proxy_config": true,
        "tls_mode": 0,
        "force_passive_ip": "",
        "client_auth_type": 0
      }
    ],
    "banner": "",
    "banner_file": "",
    "active_transfers_port_non_20": true,
    "passive_port_range": {
      "start": 50000,
      "end": 50100
    }

Here, I want to turn on the FTP service based on default protocol port (here 21).
I also want to display a login message (stored in /etc/sftpgo/banner.msg), when a user is connected to my FTP.

        "port": 21,

	"banner_file": "/etc/sftpgo/banner.msg",

Feel free to modify these settings as you want.

-The HTTPD section: It will allow you to define a port for the WebUI.

  "httpd": {
    "bindings": [
      {
        "port": 8080,
        "address": "127.0.0.1",
        "enable_web_admin": true,
        "enable_https": false,
        "client_auth_type": 0
      }
    ],
    "templates_path": "/usr/share/sftpgo/templates",
    "static_files_path": "/usr/share/sftpgo/static",
    "backups_path": "/srv/sftpgo/backups",
    "certificate_file": "",
    "certificate_key_file": "",
    "ca_certificates": [],
    "ca_revocation_lists": []
  },

Let's modify the default WebUI port to 8081.

	"port": 8081,

-The Telemetry section: It will allow you to define a port for the telemetries services.

  "telemetry": {
    "bind_port": 10000,
    "bind_address": "127.0.0.1",
    "enable_profiler": false,
    "auth_user_file": "",
    "certificate_file": "",
    "certificate_key_file": ""
  },

For some reasons, I never figured out, how to use SFTPGo with the default port which is 10000. I've used to modify it for 10001, and it works like a charm.

	"bind_port": 10001,

Optionnal part:
Last thing will be to generate SSL certificates, expose our service on 443 port.
To do that, let's use Certbot. As a reminder, to get an SSL certificate, please add an A record pointing to your Instance/VPS IPv4.

snap install core
# Here, we will use the latest packet manager from Canonical
snap install --classic certbot
# Let's install Certbot
ln -s /snap/bin/certbot /usr/bin/certbot
# Create a symlink 

Once it's done, let's generate our SSL certificate:

certbot certonly
# When asked, select "spin up a temporary webserver" option

Now, the SSL certificate is created, let's add it to SFTPGo.

cp /etc/letsencrypt/live/yourdomain.tld/cert.pem /etc/sftpgo/ssl/
cp /etc/letsencrypt/live/yourdomain.tld/privkey.pem /etc/sftpgo/ssl/
# Replace yourdomain.tld by your domain or subdomain used
chown -R sftpgo:sftpgo /etc/sftpgo/ssl

Now, all files are ready to be used by SFTPGo. Let's specify these files (mandatory) and 443 port (optional) on the sftpgo.json file.

  "httpd": {
    "bindings": [
      {
        "port": 443,
        "address": "127.0.0.1",
        "enable_web_admin": true,
        "enable_https": true,
        "client_auth_type": 0
      }
    ],
    "templates_path": "/usr/share/sftpgo/templates",
    "static_files_path": "/usr/share/sftpgo/static",
    "backups_path": "/srv/sftpgo/backups",
    "certificate_file": "/etc/sftpgo/ssl/cert.pem",
    "certificate_key_file": "/etc/sftpgo/ssl/privkey.pem",
    "ca_certificates": [],
    "ca_revocation_lists": []
  },

Enable SSL certificate by setting "enable_https" setting to "true" and add both SSL files path.

That's it, you've correctly added an SSL certificate at your SFTPGo setup!

Step 3: Start SFTPGo, and create your FTP user backed by S3

We're almost done! Let's launch SFTPGo, and see if our configuration is working.

sftpgo serve

In most of the cases, you won't get any output. And that's a good sign.

Let's check if your WebUI is online! Go to your Instance/VPS IP, and specify the port that we've modified previously. (Or don't specify port if you've chosen 443). As you can see below, that's working well!

By default, credentials are: admin / password.
Please, after your first login, modify the admin password!

Once you're logged, you will see the following dashboard. With this WebUI, you will be able to check all of your active services on the "Status" page, manage your users, folders, and administrators account.

On our case, we want to create a user called "johndoe" backed by an Scaleway S3 Bucket. Let's create it! Click on "Users", and "Add".

Fill in the "username", "password" and select wanted permissions (* = all) fields, and other ones are not mandatory.
Scroll down, and have a look at the "Storage" menu. Click on it, and select "AWS S3 (Compatible)".

Let's suppose that you've already an existing bucket at your favorite S3 provider.
In my case, I've created a bucket called "johndoebucket", in the region "fr-par", I've requested my access key and secret as explained here, and I have the endpoint "https://s3.fr-par.scw.cloud".

Then, you will have to fill in following fields:
-bucket name (here: johndoebucket)
-region (here: fr-par)
-access key (here: XXX)
-access secret (here: XXX)
-storage class (for Scaleway, choose between STANDARD or GLACIER)
-endpoint (here: https://s3.fr-par.scw.cloud)

Then, you will have a profile like this one. Now, click on "submit" to create it.

Once you've submitted the profile creation, you will be redirected to your SFTPGo dashboard.

Then, let's try to connect to our FTP server, with your favorite FTP client.

Now, you can see that we're successfully connected, and with the connected answer of the server (220), we have our banner message "Welcome to my FTP server", which is stored in /etc/sftpgo/banner.msg.

Now, let's upload a random file, and see if it's automatically upload to Scaleway S3 compatible Object Storage.

If we check on our S3 bucket, with the Scaleway built-in file explorer, or with another S3 compatible client, we will see that the object is now successfully uploaded.

Congratulations, you've successfully deployed your own FTP server backed by an S3 bucket!