drakkan/sftpgo

Fully featured and highly configurable SFTP server with optional HTTP/S, FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob

SFTPGo

Fully featured and highly configurable SFTP server with optional HTTP/S, FTP/S and WebDAV support. Several storage backends are supported: local filesystem, encrypted local filesystem, S3 (compatible) Object Storage, Google Cloud Storage, Azure Blob Storage, SFTP.

Sponsors

If you find SFTPGo useful please consider supporting this Open Source project.

Maintaining and evolving SFTPGo is a lot of work - easily the equivalent of a full time job - for me.

I'd like to make SFTPGo into a sustainable long term project and would not like to introduce a dual licensing option and limit some features to the proprietary version only.

If you use SFTPGo, it is in your best interest to ensure that the project you rely on stays healthy and well maintained. This can only happen with your donations and sponsorships ❤️

You can also purchase, using many payment methods, support plans from the SFTPGo website.

With sponsorships/donations or support plans we establish a channel for reciprocal access, ensuring better outcomes for both you and the project.

Thank you to our sponsors

Platinum sponsors

Silver sponsors

Bronze sponsors

Support policy

SFTPGo is an Open Source project and you can of course use it for free but please don't ask for free support as well.

We will check the reported issues to see if you are experiencing a bug and if so, it may or may not be fixed, we only provide support to project sponsors/donors.

If you report an invalid issue or ask for step-by-step support, your issue will remain open with no answer or will be closed as invalid without further explanation. Thanks for understanding.

Features

• Support for serving local filesystem, encrypted local filesystem, S3 Compatible Object Storage, Google Cloud Storage, Azure Blob Storage or other SFTP accounts over SFTP/SCP/FTP/WebDAV.
• Virtual folders are supported: a virtual folder can use any of the supported storage backends. So you can have, for example, a user with the S3 backend mapping a GCS bucket (or part of it) on a specified path and an encrypted local filesystem on another one. Virtual folders can be private or shared among multiple users, for shared virtual folders you can define different quota limits for each user.
• Configurable custom commands and/or HTTP hooks on upload, pre-upload, download, pre-download, delete, pre-delete, rename, mkdir, rmdir on SSH commands and on user add, update and delete.
• Virtual accounts stored within a "data provider".
• SQLite, MySQL, PostgreSQL, CockroachDB, Bolt (key/value store in pure Go) and in-memory data providers are supported.
• Chroot isolation for local accounts. Cloud-based accounts can be restricted to a certain base path.
• Per-user and per-directory virtual permissions, for each path you can allow or deny: directory listing, upload, overwrite, download, delete, rename, create directories, create symlinks, change owner/group/file mode and modification time.
• REST API for users and folders management, data retention, backup, restore and real time reports of the active connections with possibility of forcibly closing a connection.
• The Event Manager allows to define custom workflows based on server events or schedules.
• Web based administration interface to easily manage users, folders and connections.
• Web client interface so that end users can change their credentials, manage and share their files in the browser.
• Public key and password authentication. Multiple public keys per-user are supported.
• SSH user certificate authentication.
• Keyboard interactive authentication. You can easily setup a customizable multi-factor authentication.
• Partial authentication. You can configure multi-step authentication requiring, for example, the user password after successful public key authentication.
• Per-user authentication methods.
• Two-factor authentication based on time-based one time passwords (RFC 6238) which works with Authy, Google Authenticator, Microsoft Authenticator and other compatible apps.
• LDAP/Active Directory authentication using a plugin.
• Simplified user administrations using groups.
• Roles allow to create limited administrators who can only create and manage users with their role.
• Custom authentication via external programs/HTTP API.
• Web Client and Web Admin user interfaces support OpenID Connect authentication and so they can be integrated with identity providers such as Keycloak. You can find more details here.
• Data At Rest Encryption.
• Dynamic user modification before login via external programs/HTTP API.
• Quota support: accounts can have individual disk quota expressed as max total size and/or max number of files.
• Bandwidth throttling, with separate settings for upload and download and overrides based on the client's IP address.
• Data transfer bandwidth limits, with total limit or separate settings for uploads and downloads and overrides based on the client's IP address. Limits can be reset using the REST API.
• Per-protocol rate limiting is supported and can be optionally connected to the built-in defender to automatically block hosts that repeatedly exceed the configured limit.
• Per-user maximum concurrent sessions.
• Per-user and global IP filters: login can be restricted to specific ranges of IP addresses or to a specific IP address.
• Per-user and per-directory shell like patterns filters: files can be allowed, denied and optionally hidden based on shell like patterns.
• Automatically terminating idle connections.
• Automatic blocklist management using the built-in defender.
• Geo-IP filtering using a plugin.
• Atomic uploads are configurable.
• Per-user files/folders ownership mapping: you can map all the users to the system account that runs SFTPGo (all platforms are supported) or you can run SFTPGo as root user and map each user or group of users to a different system account (*NIX only).
• Support for Git repositories over SSH.
• SCP and rsync are supported.
• FTP/S is supported. You can configure the FTP service to require TLS for both control and data connections.
• WebDAV is supported.
• ACME protocol is supported. SFTPGo can obtain and automatically renew TLS certificates for HTTPS, WebDAV and FTPS from Let's Encrypt or other ACME compliant certificate authorities, using the HTTP-01 or TLS-ALPN-01 challenge types.
• Two-Way TLS authentication, aka TLS with client certificate authentication, is supported for REST API/Web Admin, FTPS and WebDAV over HTTPS.
• Per-user protocols restrictions. You can configure the allowed protocols (SSH/HTTP/FTP/WebDAV) for each user.
• Prometheus metrics are supported.
• Support for HAProxy PROXY protocol: you can proxy and/or load balance the SFTP/SCP/FTP service without losing the information about the client's address.
• Easy migration from Linux system user accounts.
• Portable mode: a convenient way to share a single directory on demand.
• SFTP subsystem mode: you can use SFTPGo as OpenSSH's SFTP subsystem.
• Performance analysis using built-in profiler.
• Configuration format is at your choice: JSON, TOML, YAML, HCL, envfile are supported.
• Log files are accurate and they are saved in the easily parsable JSON format (more information).
• SFTPGo supports a plugin system and therefore can be extended using external plugins.
• Infrastructure as Code (IaC) support using the Terraform provider.
• Partial (experimental) support for internationalization.

Platforms

SFTPGo is developed and tested on Linux. After each commit, the code is automatically built and tested on Linux, macOS, Windows and FreeBSD. Other *BSD variants should work too.

Requirements

• Go as build only dependency. We support the Go version(s) used in continuous integration workflows.
• A suitable SQL server to use as data provider:
  • upstream supported versions of PostgreSQL, MySQL and MariaDB.
  • CockroachDB stable.
• The SQL server is optional: you can choose to use an embedded SQLite, bolt or in memory data provider.

Installation

Binary releases for Linux, macOS, and Windows are available. Please visit the releases page.

An official Docker image is available. Documentation is here.

Some Linux distro packages are available

• For Arch Linux via AUR:
  • sftpgo. This package follows stable releases. It requires git, gcc and go to build.
  • sftpgo-bin. This package follows stable releases downloading the prebuilt linux binary from GitHub. It does not require git, gcc and go to build.
  • sftpgo-git. This package builds and installs the latest git main branch. It requires git, gcc and go to build.
• Deb and RPM packages are built after each commit and for each release.
• For Ubuntu a PPA is available here.
• Void Linux provides an official package.

APT and YUM repositories are available.

SFTPGo is also available on some marketplaces:

• AWS Marketplace
• Azure Marketplace: SFTPGo for Linux, SFTPGo for Windows, SFTPGo for AKS.
• Elest.io

Purchasing from there will help keep SFTPGo a long-term sustainable project.

Windows packages

• The Windows installer to install and run SFTPGo as a Windows service.
• The portable package to start SFTPGo on demand.
• The winget package to install and run SFTPGo as a Windows service: winget install SFTPGo.
• The Chocolatey package to install and run SFTPGo as a Windows service.

On macOS you can install from the Homebrew Formula. On FreeBSD you can install from the SFTPGo port. On DragonFlyBSD you can install SFTPGo from DPorts.

You can easily test new features selecting a commit from the Actions page and downloading the matching build artifacts for Linux, macOS or Windows. GitHub stores artifacts for 90 days.

Alternately, you can build from source.

Getting Started Guide for the Impatient.

Configuration

A full explanation of all configuration methods can be found here.

Please make sure to initialize the data provider before running the daemon.

To start SFTPGo with the default settings, simply run:

sftpgo serve

Check out this documentation if you want to run SFTPGo as a service.

Data provider initialization and management

Before starting the SFTPGo server please ensure that the configured data provider is properly initialized/updated.

For PostgreSQL, MySQL and CockroachDB providers, you need to create the configured database. For SQLite, the configured database will be automatically created at startup. Memory and bolt data providers do not require an initialization but they could require an update to the existing data after upgrading SFTPGo.

SFTPGo will attempt to automatically detect if the data provider is initialized/updated and if not, will attempt to initialize/ update it on startup as needed.

Alternately, you can create/update the required data provider structures yourself using the initprovider command.

For example, you can simply execute the following command from the configuration directory:

sftpgo initprovider

Take a look at the CLI usage to learn how to specify a different configuration file:

sftpgo initprovider --help

You can disable automatic data provider checks/updates at startup by setting the update_mode configuration key to 1.

You can also reset your provider by using the resetprovider sub-command. Take a look at the CLI usage for more details:

sftpgo resetprovider --help

⚠️ Please note that some data providers (e.g. MySQL and CockroachDB) do not support schema changes within a transaction, this means that you may end up with an inconsistent schema if migrations are forcibly aborted. CockroachDB doesn't support database-level locks, so make sure you don't execute migrations concurrently.

Create the first admin

To start using SFTPGo you need to create an admin user, you can do it in several ways:

• by using the web admin interface. The default URL is http://127.0.0.1:8080/web/admin
• by loading initial data
• by enabling create_default_admin in your configuration file and setting the environment variables SFTPGO_DEFAULT_ADMIN_USERNAME and SFTPGO_DEFAULT_ADMIN_PASSWORD

Upgrading

SFTPGo supports upgrading from the previous release branch to the current one. Some examples for supported upgrade paths are:

• from 2.1.x to 2.2.x
• from 2.2.x to 2.3.x and so on.

For supported upgrade paths, the data and schema are migrated automatically when SFTPGo starts, alternatively you can use the initprovider command before starting SFTPGo.

So if, for example, you want to upgrade from 2.0.x to 2.2.x, you must first install version 2.1.x, update the data provider (automatically, by starting SFTPGo or manually using the initprovider command) and finally install the version 2.2.x. It is recommended to always install the latest available minor version, ie do not install 2.1.0 if 2.1.2 is available.

Loading data from a provider independent JSON dump is supported from the previous release branch to the current one too. After upgrading SFTPGo it is advisable to regenerate the JSON dump from the new version.

Downgrading

If for some reason you want to downgrade SFTPGo, you may need to downgrade your data provider schema and data as well. You can use the revertprovider command for this task.

As for upgrading, SFTPGo supports downgrading from the previous release branch to the current one.

So, if you plan to downgrade from 2.3.x to 2.2.x, before uninstalling 2.3.x version, you can prepare your data provider executing the following command from the configuration directory:

sftpgo revertprovider

Take a look at the CLI usage to learn how to specify a configuration file:

sftpgo revertprovider --help

The revertprovider command is not supported for the memory provider.

Please note that we only support the current release branch and the current main branch, if you find a bug it is better to report it rather than downgrading to an older unsupported version.

Users, groups, folders and other resource management

After starting SFTPGo you can manage users, groups, folders and other resources using:

• the WebAdmin UI
• the REST API

To support embedded data providers like bolt and SQLite, which do not support concurrent connections, we can't have a CLI that directly write users and other resources to the data provider, we always have to use the REST API.

Full details for users, groups, folders, admins and other resources are documented in the OpenAPI schema. If you want to render the schema without importing it manually, you can explore it on Stoplight.

⚠️ SFTPGo users, groups and folders are virtual and therefore unrelated to the system ones. There is no need to create system-wide users and groups.

Tutorials

Some step-to-step tutorials can be found inside the source tree howto directory.

Authentication options

External Authentication

Custom authentication methods can easily be added. SFTPGo supports external authentication modules, and writing a new backend can be as simple as a few lines of shell script. More information can be found here.

Keyboard Interactive Authentication

Keyboard interactive authentication is, in general, a series of questions asked by the server with responses provided by the client. This authentication method is typically used for multi-factor authentication.

More information can be found here.

Dynamic user creation or modification

A user can be created or modified by an external program just before the login. More information about this can be found here.

Custom Actions

SFTPGo allows you to configure custom commands and/or HTTP hooks to receive notifications about file uploads, deletions and several other events.

More information about custom actions can be found here.

Virtual folders

Directories outside the user home directory or based on a different storage provider can be mapped as virtual folders, more information here.

Other hooks

You can get notified as soon as a new connection is established using the Post-connect hook and after each login using the Post-login hook. You can use your own hook to check passwords.

Storage backends

S3/GCP/Azure

Each user can be mapped with a S3 Compatible Object Storage /Google Cloud Storage/Azure Blob Storage bucket or a bucket virtual folder.

SFTP backend

Each user can be mapped to another SFTP server account or a subfolder of it. More information can be found here.

Encrypted backend

Data at-rest encryption is supported via the cryptfs backend.

HTTP/S backend

HTTP/S backend allows you to write your own custom storage backend by implementing a REST API. More information can be found here.

Other Storage backends

Adding new storage backends is quite easy:

• implement the Fs interface.
• update the user method GetFilesystem to return the new backend
• update the web interface and the REST API CLI
• add the flags for the new storage backed to the portable mode

Anyway, some backends require a pay per-use account (or they offer free account for a limited time period only). To be able to add support for such backends or to review pull requests, please provide a test account. The test account must be available for enough time to be able to maintain the backend and do basic tests before each new release.

Brute force protection

SFTPGo supports a built-in defender.

Alternately you can use the connection failed logs for integration in tools such as Fail2ban. Example of jails and filters working with systemd/journald are available in fail2ban directory.

Account's configuration properties

Details information about account configuration properties can be found here.

Performance

SFTPGo can easily saturate a Gigabit connection on low end hardware with no special configuration, this is generally enough for most use cases.

More in-depth analysis of performance can be found here.

Release Cadence

SFTPGo releases are feature-driven, we don't have a fixed time based schedule. As a rough estimate, you can expect 1 or 2 new releases per year.

Acknowledgements

SFTPGo makes use of the third party libraries listed inside go.mod.

We are very grateful to all the people who contributed with ideas and/or pull requests.

Thank you to ysura for granting us stable access to a test AWS S3 account.

Thank you to KeenThemes for granting us a custom license to use their amazing Mega Bundle for SFTPGo UI.

Thank you to Crowdin for granting us an Open Source License.

Thank you to Incode for helping us to improve the UI/UX.

License

GNU AGPL-3.0-only

The theme used in WebAdmin and WebClient user interfaces is proprietary, this means:

• KeenThemes HTML/CSS/JS components are allowed for use only within the SFTPGo product and restricted to be used in a resealable HTML template that can compete with KeenThemes products anyhow.
• The SFTPGo WebAdmin and WebClient user interfaces (HTML, CSS and JS components) based on this theme are allowed for use only within the SFTPGo product and therefore cannot be used in derivative works/products without an explicit grant from the SFTPGo Team.

More information about compliance.

sftpgo's Issues

Switch to viper?

Similar as my kingpin suggestion, viper is a great tool for configuration loading. It supports multiple formats out of the box, can go through multiple loading locations (eg. [$HOME, "/etc/sftpgo/"]) and has numerous interesting features.

I would also suggest to use sftpgo.json instead of sftpgo.conf for the json-formatted configuration file - my editor would be grateful :-)

Detect errored upload ?

I have a use case where I want detect broken upload.
I mean i send a file via scp I get the action. this is fine.
I send a file via scp I kill scp. Today the server will fire an action the same way.

Is there any way to detect those ?

suggestion: Windows Service restart behavior

SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z
Windows

When you run sftpgo.exe service install, it creates a Windows service with a restart behavior of "always restart". I suggest you change that default behavior to only restart twice, and then if it fails a third time, let it stay stopped. The other Issue I created demonstrates a reason for this: it keeps restarting over and over, and crashes every time. It would be better if it gave up and quit wasting resources. It's doubtful that a service that crashes twice in a row will be successful the third time, so please don't tell it to restart infinitely, just twice.

Some issues with rclone

Hi,

I'm trying to use rclone but ran into some issues.

Path with space and symbol not correctly handled by checksum commands

For example:
/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Brahms Symphony No.4 Carlos Kreiber Vienna Philharmonic (SHM-CD)/albumart.pamp

rclone generate command:
sftp cmd = /music/DG/C.kleiber\ Originals\ Box-dg\ Recordings\(SHM-CD\)/Brahms\ Symphony\ No.4\ \ Carlos\ Kreiber\ \ Vienna\ Philharmonic\ \(SHM-CD\)/albumart.pamp

In sftpgo's log:

{"level":"debug","time":"2020-02-14T09:47:08.656","sender":"ssh","connection_id":"131d0931bfd5a9069d26124580e4442fb4bdc4fe9d7a044370f414babe2ed0f0","message":"new ssh command: \"md5sum\" args: [/music/DG/C.kleiber Originals Box-dg Recordings (SHM-CD )/Brahms Symphony No.4  Carlos Kreiber  Vienna Philharmonic  (SHM-CD )/albumart.pamp] user: ftp_public, error: <nil>"}
{"level":"warn","time":"2020-02-14T09:47:08.656","sender":"ssh","connection_id":"131d0931bfd5a9069d26124580e4442fb4bdc4fe9d7a044370f414babe2ed0f0","message":"command failed: \"md5sum\" args: [/music/DG/C.kleiber Originals Box-dg Recordings (SHM-CD )/Brahms Symphony No.4  Carlos Kreiber  Vienna Philharmonic  (SHM-CD )/albumart.pamp] user: ftp_public err: open /media/public/music/DG/C.kleiber Originals Box-dg Recordings (SHM-CD )/Brahms Symphony No.4  Carlos Kreiber  Vienna Philharmonic  (SHM-CD )/albumart.pamp: no such file or directory"}

It seems like escape of symbols, eg.\( is also translated into space. It's also reproducible by manually enter md5sum command.

Transfer error (SSH_FX_FAILURE)

Not much information provided by the log.
rclone:

2020/02/14 09:46:53 DEBUG : C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac: multi-thread copy: stream 1/2 (0-171573248) size 163.625M finished
2020/02/14 09:46:53 DEBUG : C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac: multi-thread copy: stream 1/2 failed: sftp: "incomplete download: 173670400/343043570 bytes transferred" (SSH_FX_FAILURE)
2020/02/14 09:46:53 DEBUG : C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac: multi-thread copy: stream 2/2 failed: context canceled
2020/02/14 09:46:53 ERROR : C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac: Failed to copy: sftp: "incomplete download: 173670400/343043570 bytes transferred" (SSH_FX_FAILURE)

sftpgo:

{"level":"debug","time":"2020-02-14T10:45:14.386","sender":"sftpd","connection_id":"b1adc368547b89c4a788affb0d92d392bd090d9341b1a39930f511851e597989","message":"fileread requested for path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}
{"level":"debug","time":"2020-02-14T10:45:14.386","sender":"sftpd","connection_id":"73148db23ebefb1ad78cb41e43b302281a2d7813f5909e5d8f18bb1befae4901","message":"fileread requested for path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}
{"level":"debug","time":"2020-02-14T10:45:14.387","sender":"sftpd","connection_id":"b1adc368547b89c4a788affb0d92d392bd090d9341b1a39930f511851e597989","message":"requested stat for path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}
{"level":"debug","time":"2020-02-14T10:45:14.388","sender":"sftpd","connection_id":"73148db23ebefb1ad78cb41e43b302281a2d7813f5909e5d8f18bb1befae4901","message":"requested stat for path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}
{"level":"warn","time":"2020-02-14T10:45:35.825","sender":"sftpd","connection_id":"73148db23ebefb1ad78cb41e43b302281a2d7813f5909e5d8f18bb1befae4901","message":"transfer error: incomplete download: 173670400/343043570 bytes transferred, path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}
{"level":"warn","time":"2020-02-14T10:45:36.190","sender":"sftpd","connection_id":"b1adc368547b89c4a788affb0d92d392bd090d9341b1a39930f511851e597989","message":"transfer error: incomplete download: 161316864/343043570 bytes transferred, path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}

Environmental variable configuration for data_provider not working (user error?)

Is this the correct syntax? When I do not provide a configuration file and instead set these env variables it doesn't seem to be applying those values.

SFTPGO_DATA_PROVIDER__DRIVER=mysql
SFTPGO_DATA_PROVIDER__NAME=mydb
SFTPGO_DATA_PROVIDER__HOST=127.0.0.1
SFTPGO_DATA_PROVIDER__PORT=330

sftp    | {"level":"info","sender":"cmd","time":"2019-09-03T20:47.33.437","message":"starting SFTPGo, config dir: ., config file: sftpgo, log max size: 10 log max backups: 0 log max age: 0 log verbose: true, log compress: false"}
sftp    | {"level":"warn","sender":"config","time":"2019-09-03T20:47.33.438","message":"error loading configuration file: Config File \"sftpgo\" Not Found in \"[/app/.config/sftpgo /etc/sftpgo /app]\". Default configuration will be used: {SFTPD:{Banner:SFTPGo BindPort:2022 BindAddress: IdleTimeout:15 MaxAuthTries:0 Umask:0022 UploadMode:0 Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} Keys:[] IsSCPEnabled:false} ProviderConf:{Driver:sqlite Name:sftpgo.db Host: Port:5432 Username: Password: SSLMode:0 ConnectionString: UsersTable:users ManageUsers:1 TrackQuota:1} HTTPDConfig:{BindPort:8080 BindAddress:127.0.0.1}}"}
sftp    | 2019-09-03T20:47.33.438 WRN error loading configuration file: Config File "sftpgo" Not Found in "[/app/.config/sftpgo /etc/sftpgo /app]". Default configuration will be used.
sftp    | {"level":"warn","sender":"dataProvider","time":"2019-09-03T20:47.33.438","message":"sqlite database file does not exists, please be sure to create and initialize a database before starting sftpgo"}
sftp    | {"level":"error","sender":"cmd","time":"2019-09-03T20:47.33.438","message":"error initializing data provider: stat sftpgo.db: no such file or directory"}
sftp    | 2019-09-03T20:47.33.438 ERR error initializing data provider: stat sftpgo.db: no such file or directory
sftp exited with code 1

Feature suggest: Let the API create a UNIX user

Hi

Can it be possible (and it's a huge security risk) to have the API to create a new UNIX user?

It's useful for when you want to use the uid/guid parameters to have more granular control of the folders in the users home folder.

Support simple config file as data provider (for users definition)

Hi,
You have started a very nice project ^_^
I would like to run SFTPGo in a higly secure environment or at least ask security team for the permission ^^'

I'm looking for a file server (protocol can be sftp, or else) with virtual user / non unix user, with everything in immutable configuration file.
Config can be generated by a config manager like chef, ansible watever.
You already have conf key to disable web interface etc.
So, I suggest a config file as data provider.

Have a nice day !

Relatively lower performance than OpenSSH

Hi,
Thanks for this great project !

I did some test in my environment and the transfer speed is much lower than OpenSSH.

Server
OS	Debian 10.2 x64
CPU	Ryzen5 3600
RAM	64GB ECC
Disk	3* Intel P4510 4TB RAID0
Ethernet	Mellanox ConnectX-3 40GbE

Client
OS	Windows 10 1909 x64
CPU	Threadripper 1920X
RAM	64GB ECC
Disk	Samsung 960EVO 1TB
Ethernet	Mellanox ConnectX-3 40GbE

Under Filezilla I can get 500MB/s with OpenSSH, but only about 200MB/s with sftpgo.

In both case I'm using AES256-CTR as cipher and SHA-256 as MAC, I've also tried AES128-CTR but nothing changes.

CPU usage of sftpgo is higher than OpenSSH:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND 
 4527 sftp      20   0 1795576  52044   8628 R 133.5   0.6   2:12.13 sftpgo 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND 
27934 xxxxxx    20   0   17112   5360   4188 R  67.8   0.1   0:10.01 sshd                                                  
27942 xxxxxx    20   0   17112   5344   4176 R  27.4   0.1   0:12.52 sshd 

In both case I've got a maximum TCP window size of 4MB.

Support for Logging to stdout/stderr

Hello!

We're using SFTPGo in a Docker environment, and while the flatfile logging option isn't particularly difficult for us to work around (using other tools like Dockerize), it's always much easier if we can just pass output directly to stdout/stderr, where the Docker stack can manage it according to its own log rotation rules.

Would this be rather straightforward to support in the app?

quota_files crashes sftpgo

When I create a user with the quota_files parameter set to 5 for example and then try to upload any file after the quota is reached it will crash sftpgo.

Here is the crash log from syslog:

Jul 26 11:50:38 raspberrypi systemd[1]: Started SFTPGo sftp server.
Jul 26 11:50:53 raspberrypi sftpgo[12844]: panic: runtime error: invalid memory address or nil pointer dereference
Jul 26 11:50:53 raspberrypi sftpgo[12844]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x43be40]
Jul 26 11:50:53 raspberrypi sftpgo[12844]: goroutine 86 [running]:
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/drakkan/sftpgo/dataprovider.GetUsedQuota(...)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/drakkan/sftpgo/dataprovider/dataprovider.go:132
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/drakkan/sftpgo/sftpd.Connection.hasSpace(0x286e540, 0x40, 0x8, 0x0, 0x28885d8, 0x4, 0x29ca2a0, 0x61, 0x0, 0x0, ...)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/drakkan/sftpgo/sftpd/handler.go:404 +0x380
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/drakkan/sftpgo/sftpd.Connection.Filewrite(0x286e540, 0x40, 0x8, 0x0, 0x28885d8, 0x4, 0x29ca2a0, 0x61, 0x0, 0x0, ...)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/drakkan/sftpgo/sftpd/handler.go:98 +0x984
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/pkg/sftp.(*Request).open(0x2932060, 0x66f540, 0x29b0240, 0x66f558, 0x29b0300, 0x66f510, 0x29b03c0, 0x66f528, 0x29b0480, 0x6715e0, ...)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/pkg/sftp/request.go:180 +0x114
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/pkg/sftp.(*RequestServer).packetWorker(0x2942500, 0x673520, 0x2902a60, 0x286e5c0, 0x0, 0x0)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/pkg/sftp/request-server.go:167 +0x4b8
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/pkg/sftp.(*RequestServer).Serve.func1.1(0x2888630, 0x2942500, 0x673520, 0x2902a60, 0x286e5c0)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/pkg/sftp/request-server.go:98 +0x58
Jul 26 11:50:53 raspberrypi sftpgo[12844]: created by github.com/pkg/sftp.(*RequestServer).Serve.func1
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/pkg/sftp/request-server.go:96 +0x7c
Jul 26 11:50:53 raspberrypi systemd[1]: sftpgo.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 26 11:50:53 raspberrypi systemd[1]: sftpgo.service: Failed with result 'exit-code'.

httpd ignoring conf file?

Hello,

My conf file is very similar to your example, except I wanted to test out the httpd API and made the changes below.

   "httpd":{
       "bind_port":8000,
           "bind_address": ""
   }

1. Even though the port has changed to 8000, nothing answers there and still wants to answer on 8080. Tested with nc on the localhost and remote host.

2. Even though I have to listening on all available addresses, nothing answers on remote hosts. Tested with a webbrowser and nc on two separate hosts.

Am I doing something wrong? A missing comma, period, quotes?

Thanks!

portable mode sftpgo.db error

I build sftpgo docker image with alpine instructions.

I am trying to start server with sftpgo portable, but it throws ERR error initializing data provider: stat sftpgo.db: no such file or directory error all the time.

I investigated the code and portable mode selects memory data provider. Unfortunately I could not find the bug.

Connecting with WinScp

Hi,

Sftpgo is running fine on my ubuntu server. (the log is fine)

I have manually inserted a user with a password in the sqlite database.
While connecting via WinScp it throws a authentication error.
Hope I am doing the right steps?

Thanks,

Ajay

Include SystemD init file in release

I'm building a pipeline to release an RPM based on your versioned releases (thanks for that!); I'm missing the SystemD init file. I believe this should be included as well?

Support with S3 integration

I have some questions about the s3 configuration parameters listed in the doc.

• s3 endpoint is always required? if it is, you are referring to an s3 access point?

zombie processes

I'm using the actions configuration to run a callback upon uploads and it appears the process is not being properly cleaned up. I was running it as pid 1, but switched to tini to see if that would help, but it doesn't.

I'm running this in docker and/or kubernetes inside an alpine image.

Here is an example of ps after I upload two files. The tester is the command that is run in the action. You can see the stat is set to Z.

~ $ ps -o pid,vsz,rss,tty,stat,time,ruser,args
PID   VSZ  RSS  TT     STAT TIME  RUSER    COMMAND
    1  768    4 ?      S     0:00 sftp     /sbin/tini -- /app/sftpgo serve
    6 248m 145m ?      S     0:01 sftp     /app/sftpgo serve
   13 1628  992 136,0  S     0:00 sftp     ash
   27    0    0 ?      Z     0:00 sftp     [tester]
   29    0    0 ?      Z     0:00 sftp     [tester]
   30 1556    4 136,0  R     0:00 sftp     ps -o pid,vsz,rss,tty,stat,time,ruser,args

Here is the Dockerfile

FROM golang:1.12-alpine3.10 as sftpgo
RUN apk add --no-cache git gcc g++ ca-certificates \
  && go get -u github.com/drakkan/sftpgo

FROM alpine:3.10
RUN  apk add --no-cache openssh-keygen curl tini \
  && adduser sftp -h /app -D \
  && mkdir -p /data /app/log \
  && ln -sf /dev/stdout /app/log/sftpgo.log \
  && ssh-keygen -t rsa -N "" -f /app/id_rsa \
  && chown sftp /data /app/id_rsa /app/id_rsa.pub /app/log
COPY --from=sftpgo /go/bin/sftpgo /app
COPY . /app
RUN chown sftp /app/sftpgo.db
WORKDIR /app
USER sftp
EXPOSE 2022
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["/app/sftpgo","serve"]

config

{
  "sftpd": {
    "bind_port": 2022,
    "bind_address": "",
    "idle_timeout": 15,
    "max_auth_tries": 0,
    "umask": "0022",
    "upload_mode": 1,
    "banner": "tester sftp",
    "actions": {
      "execute_on": ["upload"],
      "command": "/app/tester",
      "http_notification_url": ""
    },
    "keys": []
  },
  "data_provider": {
    "driver": "sqlite",
    "name": "sftpgo.db",
    "host": "",
    "port": 5432,
    "username": "",
    "password": "",
    "sslmode": 0,
    "connection_string": "",
    "users_table": "users",
    "manage_users": 1,
    "track_quota": 0
  },
  "httpd": {
    "bind_port": 8080,
    "bind_address": "127.0.0.1"
  }
}

action script

#!/bin/ash

ACTION=$1
USERNAME=$2
PATH=$3
TARGET_PATH=$4

echo "GOT $1 for $2 @ $3"

I haven't done much in golang to be useful here for submitting a PR.

However, I did fine some related links.

https://stackoverflow.com/questions/36050503/golang-child-processes-become-zombies
https://github.com/ramr/go-reaper

Let me know if you want some help in this and I can give it a shot.

error if trailing backslash in config-dir (Windows)

SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z

In Windows, I'd expect to type paths with backslashes:
sftpgo.exe service install --config-dir="C:\ProgramData\SFTPGO\"
but it writes an error to C:\Program Files\SFTPGo\sftpgo.log unless I use forward slashes:
sftpgo.exe service install --config-dir="C:/ProgramData/SFTPGO/"

{"level":"info","time":"2020-03-01T21:57:32.547","sender":"service","connection_id":"","message":"starting SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z, config dir: C:\\ProgramData\\SFTPGO\", config file: sftpgo, log max size: 10 log max backups: 5 log max age: 28 log verbose: true, log compress: false"}
{"level":"warn","time":"2020-03-01T21:57:32.550","sender":"config","connection_id":"","message":"error loading configuration file: Config File \"sftpgo\" Not Found in \"[C:\\\\ProgramData\\\\SFTPGO\\\" C:\\\\Program Files\\\\SFTPGo]\". Default configuration will be used: {SFTPD:{Banner:SFTPGo_0.9.5 BindPort:2022 BindAddress: IdleTimeout:15 MaxAuthTries:0 Umask:0022 UploadMode:0 Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} Keys:[] IsSCPEnabled:false KexAlgorithms:[] Ciphers:[] MACs:[] LoginBannerFile: SetstatMode:0 EnabledSSHCommands:[md5sum sha1sum cd pwd]} ProviderConf:{Driver:sqlite Name:sftpgo.db Host: Port:5432 Username: Password:[redacted] SSLMode:0 ConnectionString: UsersTable:users ManageUsers:1 TrackQuota:1 PoolSize:0 UsersBaseDir: Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} ExternalAuthProgram: ExternalAuthScope:0} HTTPDConfig:{BindPort:8080 BindAddress:127.0.0.1 TemplatesPath:templates StaticFilesPath:static BackupsPath:backups}}"}
{"level":"warn","time":"2020-03-01T21:57:32.551","sender":"sqlite","connection_id":"","message":"sqlite database file does not exists, please be sure to create and initialize a database before starting sftpgo"}
{"level":"error","time":"2020-03-01T21:57:32.551","sender":"service","connection_id":"","message":"error initializing data provider: CreateFile C:\\ProgramData\\SFTPGO\"\\sftpgo.db: The filename, directory name, or volume label syntax is incorrect."}

You'll also notice that the log file has way too many slashes in it. Looks like you're trying to escape characters using slashes, but that ends up printing them incorrectly to the log:
Config File \"sftpgo\" Not Found in \"[C:\\\\ProgramData\\\\SFTPGO\\\" C:\\\\Program Files\\\\SFTPGo]\".

Logging with public keys

Hi drakkan,

Would you have an example of how to create a user with his public key?

Maybe using the python client.

Thanks,

Ajay

Create a CLI

The REST API is great, of course, but a more intuitive CLI would also be awesome.

Two options:

• build a separate tool (in any language, but I personally would prefer golang)
• make the daemon also the CLI (see eg. docker, vault and nomad)

In the first scenario, you will need to keep both projects in sync where relevant.

In the second case, you should use a specific "command" argument to run the server, so you don't accidentally run it and create all kinds of default files (log, id_rsa)

eg.:

$ sftpgo serve --config-dir=/etc/sftpgo/
$ sftpgo user create foo --uid=1000 --homedir=/var/lib/sftpgo/foo --keys=@/path/to/authorized_keys --no-password

Occasional Malfunction of Connection

Occasionally (about once every 12-18 hours) the server will stop accepting the correct username and password. It requires a service reboot to fix. This is the closest set of logs to the latest attempt to login while in this error state:

{"level":"debug","sender":"sftpd","connection_id":"","time":"2019-09-11T03:27.05.832","message":"idle connections check ticker 2019-09-11 03:27:05.832582724 +0000 UTC m=+44100.001673267"}
{"level":"info","sender":"sftpd","connection_id":"6508f97ceb9b67f8e7fc4928cd9b0cd10615a47024a8279611e9e8f3f6ef24b4","time":"2019-09-11T03:27.05.832","message":"close idle connection, idle time: 6h27m4.548987039s"}
{"level":"warn","sender":"sftpd","connection_id":"6508f97ceb9b67f8e7fc4928cd9b0cd10615a47024a8279611e9e8f3f6ef24b4","time":"2019-09-11T03:27.05.832","message":"idle connection close failed: close tcp 172.31.18.230:22->148.59.44.16:64168: use of closed network connection"}
{"level":"info","sender":"sftpd","connection_id":"8fe250f6489ada67fcb4161871743f4358892324ce8841dcbe4903eabfcc2a59","time":"2019-09-11T03:27.05.832","message":"close idle connection, idle time: 6h27m4.535060735s"}
{"level":"warn","sender":"sftpd","connection_id":"8fe250f6489ada67fcb4161871743f4358892324ce8841dcbe4903eabfcc2a59","time":"2019-09-11T03:27.05.832","message":"idle connection close failed: close tcp 172.31.18.230:22->148.59.44.16:64169: use of closed network connection"}
{"level":"debug","sender":"sftpd","connection_id":"","time":"2019-09-11T03:27.05.832","message":"check idle connections ended"}

Any ideas? Is there any way to stop it trying to close idle connections? Perhaps that's the problem?

Any clients ?

A golang cli client ?
Can build a GUI client on top of the cli client too I think.

Let me know what's lacking .. :)

initprovider missing in Windows build

SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z

New user, I see this error in the log:

{"level":"warn","time":"2020-03-01T21:59:58.969","sender":"sqlite","connection_id":"","message":"sqlite database file does not exists, please be sure to create and initialize a database before starting sftpgo"}
{"level":"error","time":"2020-03-01T21:59:58.969","sender":"service","connection_id":"","message":"error initializing data provider: CreateFile C:\\ProgramData\\SFTPGO\\sftpgo.db: The system cannot find the file specified."}

The readme says "For SQLite provider the database file will be auto created if missing" so I thought I wouldn't need to run initprovider. Seeing the error though, I ran initprovider as the readme instructs, but the executable does not recognize initprovider.

C:\Program Files\SFTPGo>sftpgo initprovider
Error: unknown command "initprovider" for "sftpgo"
Run 'sftpgo --help' for usage.
unknown command "initprovider" for "sftpgo"

C:\Program Files\SFTPGo>sftpgo --help
Full featured and highly configurable SFTP server

Usage:
  sftpgo [command]

Available Commands:
  help        Help about any command
  portable    Serve a single directory
  serve       Start the SFTP Server
  service     Install, Uninstall, Start, Stop and retrieve status for SFTPGo Windows Service

Flags:
  -h, --help      help for sftpgo
  -v, --version

Use "sftpgo [command] --help" for more information about a command.

I can't create the database so I'm stuck for now.

Support multiple public keys per account

It would be useful to support multiple public keys per account. Similar to authorized_keys.

Using goreleaser for binary releases

Hey there,

great project. What do you think about using goreleaser (https://goreleaser.com/) for binary releases?
I used it in some projects and it was a great tool for binary releases.

See https://github.com/sandreas/graft/blob/master/.goreleaser.yml for an example.

sandreas

can't login with public key after updating to the latest code and updating database

I just updated my test setup to the latest master, and ran the sql update line:

ALTER TABLE "users" RENAME COLUMN "public_key" TO "public_keys";

Now I can no longer login with the existing keys.

I reverted to the commit 5ad222fc53c01339b2acecd6849555b7006825b1: it works.
I cherry-pick 5ad222fc53c01339b2acecd6849555b7006825b1 and run the SQL line: it no longer works.

Should I change something else?

Console messages

sftpgo should output to console some informational messages about what it is doing. Errors such as not being able to load config file should be output to console aswell as log file.

Port 22 already in use

When attempting to start the server I get this error:

2019-07-31T15:34.16.762 ERR could not start SFTP server: listen tcp 0.0.0.0:22: bind: address already in use

How can I expose the SFTP server on port 22 when ssh is already running on that port?

Better user tracking in the logs

I have a few thoughts/requests, I can make separate issues if you think that would be better.

a) It would be great to be able to identify connecting users from the logs by the fingerprint and the comment (the last part) of the public key. Now we only see the user name.

b) I only see the user name when the user does actions. Is the user not authenticated earlier?

c) I think the logs should use the connection_id to be able to trace connections/actions.

d) Somtimes paths are quoted in the logs, sometimes not; this should obviously be consistent :-)
"requested list/stat" entries are not quoted
"fileread requested" is quoted

What I see now:

{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.56.827","message":"accepted inbound connection, ip: 10.10.10.1:4727"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.56.829","message":"connection added, num open connections: 1"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.58.496","message":"requested list file for dir: /var/ftp/myuser user: myuser"} 
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.01.497","message":"requested stat for file: /var/ftp/myuser/in user: myuser"} 
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.02.910","message":"requested list file for dir: /var/ftp/myuser/in user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.05.315","message":"requested stat for file: /var/ftp/myuser/in/somefile.xlsx user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.11.706","message":"fileread requested for path: \"/var/ftp/myuser/in/somefile.xlsx\", user: myuser"}
{"level":"info","sender":"Download","elapsed_ms":57,"size_bytes":704072,"username":"myuser","file_path":"/var/ftp/myuser/in/somefile.xlsx","connection_id":            
"77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","protocol":"SFTP","time":"2019-09-05T10:04.11.763"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.13.278","message":"connection closed, id: 8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.13.279","message":"connection removed, num open connections: 0"}

What I expect to see (more or less):

{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.56.827","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"accepted inbound connection, ip: 10.10.10.1:4727"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.56.829","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"connection added, num open connections: 1"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.56.829","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"user authenticated with public key \"jo@desktop\" fingerprint:SHA256:FV3+wlAKGzYy7+J02786fh8N8c06+jga/mdiSOSPT7g"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.58.496","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"requested list file for dir: \"/var/ftp/myuser\" user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.01.497","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"requested stat for file: \"/var/ftp/myuser/in\" user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.02.910","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"requested list file for dir: \"/var/ftp/myuser/in\" user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.05.315","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"requested stat for file: \"/var/ftp/myuser/in/somefile.xlsx\" user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.11.706","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"fileread requested for path: \"/var/ftp/myuser/in/somefile.xlsx\", user: myuser"}
{"level":"info","sender":"Download","elapsed_ms":57,"size_bytes":704072,"username":"myuser","file_path":"/var/ftp/myuser/in/somefile.xlsx","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","protocol":"SFTP","time":"2019-09-05T10:04.11.763"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.13.278","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"connection closed"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.13.279","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"connection removed, num open connections: 0"}

Fail2ban integration: Adding the source IP address in logs

Hello Drakkan,

I was thinking about fail2ban integration with sftpgo but unfortunatly, they is no the source IP address in logs.
I notify two cases :

When user does not exist:
{"level":"warn","sender":"sqlite","connection_id":"","time":"2019-11-10T20:09.25.756","message":"error authenticating user: toto, error: Not found: sql: no rows in result set"}

When user exist but password failed:
{"level":"warn","sender":"sftpd","connection_id":"","time":"2019-11-10T20:10.08.775","message":"failed to accept an incoming connection: [ssh: no auth passed yet, could not validate credentials, could not validate credentials, could not validate credentials, could not validate credentials, could not validate credentials]"}
In this second case, user is not mentioned in the log. Is it normal ?

I'm not developer, but if someone add IP in logs, i'll make the fail2ban configuration :)
Regards,

Log version when starting

After I start the sftpgo daemon, I expect it to log it's version, similar to the output from sftpgo --version.

Support for rsync

Since you now support scp, I think rsync would be a logical next step :-)

unmarshal error for field Language of type disconnectMsg

After updating to the latest version I get this error message when attempting to connect:

unmarshal error for field Language of type disconnectMsg

And then it times out. Any ideas?

API authentication

Is it possible to force authentication for the API endpoints?

Dockerfile

I am trying to Dockerize the app and trying this:

FROM golang:1.12.7-stretch
RUN mkdir /app 
ADD . /app/ 
WORKDIR /app

RUN apt-get update
RUN apt-get install -yq git-all
RUN go get -u github.com/drakkan/sftpgo
CMD ["sftpgo -config-dir /app"]

However, when I run it with:

docker run -p 8080:8080 sftpgo

I get:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"sftpgo -config-dir /app\": stat sftpgo -config-dir /app: no such file or directory": unknown.

Any ideas?

error building with latest go 1.14

Hello, there are errors building with latest go 1.14:

git pull

remote: Enumerating objects: 62, done.
remote: Counting objects: 100% (62/62), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 35 (delta 29), reused 35 (delta 29), pack-reused 0
Unpacking objects: 100% (35/35), done.
From https://github.com/drakkan/sftpgo
7163fde..833b702 master -> origin/master
Updating 7163fde..833b702

go build

github.com/drakkan/sftpgo/sftpd

sftpd/ssh_cmd.go:279:20: undefined: errors.Is
note: module requires Go 1.13

Suggestion: add dependencies in vendor dir

It seems to be a common practice now in Goland to add your dependencies in the vendor/ dir. This would mean I (as a contributor) would not need to get my own copy of your dependencies, since they live inside your git tree.

Just a suggestion.

Question: Authenticating via external script command?

Hello!

I'm trying to integrate SFTP access into an existing application which has its own user accounts and roles defined internally. Because the main application is in PHP and uses the built-in argon2i password hashing method, its passwords are hashed in a slightly different format than the one sftpgo expects, so I can't just point this app at the same database that the web application is using.

What I'm curious to find out is whether authentication via invoking a script is possible in the current version of this application. I see a number of built-in locations where hooks can be called to either programs or external URLs, including one in the data_providers section, but it doesn't appear that this hook can actually handle authentication.

Given this app already has its own internal REST API, my current plan is just to use either that REST API or a small SQLite database to sync up the web application's user accounts and the sftpgo accounts periodically.

Is that the current best solution for this kind of setup? Thanks in advance for your assistance, and a massive kudos to the team who built this wonderful application, as it solves a number of problems I've been having with my existing FTP implementation.

Feature request: Keyboard interactive auth

Hi, first of all I'd like to thank you for this amazing project. I was looking for such a simple and hackable sftp server for a very long time. Great work!

Are you interested in implementing Keyboard Interactive Authentication? It could be used to perform multi factor authentication. I think it's necessary to allow to specify custom executables in this case in order to do dynamic questions / answers handling (similar to how external auth program works now).

Support for FTPs

Many SFTP servers support the FTPS protocol too. To ensure compatibility if someone wants to use SFTPgo instead of their FTP server, it would be wise to support FTPs as well.

Take action on uploaded file

Hi there,

Thanks for this great library! Having something like this would have saved my team and I a bunch of time had it been around a year ago :)

One thing that we do that doesn't seem to be supported in this library is taking immediate action on a file that is added to the server. So when upload.csv gets added to a user's folder, we immediately pick it up, ship it off to a different server via an API, and delete the file (via a script that is pretty fragile).

Is something like that in scope for this project? I think it would be an amazing addition that could really pull this together as a fully managed solution.

Regardless of if it is possible or not, we plan to switch to this project in the near future :) Thank you for your work!!

Switch to kingpin?

For argument parsing in my golang projects, I use kingpin. Would you be interested in a PR for switching to this library?

portable/User Mode

Hi,
it would be cool, if your sftpgo server would have the following feature, a portable mode.
a user can start sftpgo, perhaps with --portable as parameter, but has at least to provide the parameter directory, where he has access.
at most he has to deliver the parameters

• password (could be generated by sftpgo as well, so optional)
• user (could be optional)
• port (has to be a non privileged port)

and this should stop when the user stops/exits the program.

Additionally (especially for mobile users) you could output an qrcode which has the sftp-url with user, port, ip/name as qr-code, perhaps as mode even with password, so mobile users could just use the qrcode to create an connection.

i would be willing to try helping with it, but i am a total newbie in go, though i wrote programs in python and perl back in the day.

Ability to choose the log processing

Hi,

I have dockerized SFTPgo service with a custom systemD script management.
It's working but i must choose a location path for the logfile with --log-file-path argument and i must have to mount a volume only for this. too bad...

Is it possible to have a parameter to change logger process and publish all logs to stdout/stderr ? and avoid to specify a log path and log file ?
With systemD and LogIdentifier parameters, it will be easy to processing logs throuth journalctl and syslog.
I can submit Dockerfile and custom SystemD script with a pull request if you want.

Thanks in advance

Feature request: add "Denied" to sub-directory permission

Maybe I've overlooked something, but it seems like there is no way to block a user form accessing a sub-directory.

Currently I'm using only listing as permission to prevent download, but user can still see the content.

crash when creating private key

SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z
Windows 10

I found a db file at C:\Program Files\SFTPGo\sftpgo.db, so I copied it to my config-dir and tried to start the windows service but the logs show the service keeps restarting when it gets to "creating new private key for server"

{"level":"info","time":"2020-03-01T22:18:11.303","sender":"service","connection_id":"","message":"starting SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z, config dir: C:/ProgramData/SFTPGO/, config file: sftpgo, log max size: 10 log max backups: 5 log max age: 28 log verbose: true, log compress: false"}
{"level":"debug","time":"2020-03-01T22:18:11.305","sender":"config","connection_id":"","message":"config file used: 'C:\\ProgramData\\SFTPGO\\sftpgo.json', config loaded: {SFTPD:{Banner:Private Server BindPort:2022 BindAddress: IdleTimeout:15 MaxAuthTries:5 Umask:0022 UploadMode:1 Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} Keys:[] IsSCPEnabled:false KexAlgorithms:[] Ciphers:[] MACs:[] LoginBannerFile: SetstatMode:0 EnabledSSHCommands:[md5sum sha1sum cd pwd]} ProviderConf:{Driver:sqlite Name:sftpgo.db Host: Port:999 Username: Password:[redacted] SSLMode:0 ConnectionString: UsersTable:users ManageUsers:1 TrackQuota:0 PoolSize:0 UsersBaseDir: Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} ExternalAuthProgram: ExternalAuthScope:0} HTTPDConfig:{BindPort:8081 BindAddress: TemplatesPath:templates StaticFilesPath:static BackupsPath:backups}}"}
{"level":"debug","time":"2020-03-01T22:18:11.305","sender":"sqlite","connection_id":"","message":"sqlite database handle created, connection string: \"file:C:\\\\ProgramData\\\\SFTPGO\\\\sftpgo.db?cache=shared\""}
{"level":"debug","time":"2020-03-01T22:18:11.306","sender":"httpd","connection_id":"","message":"initializing HTTP server with config {BindPort:8081 BindAddress: TemplatesPath:templates StaticFilesPath:static BackupsPath:backups}"}
{"level":"debug","time":"2020-03-01T22:18:11.306","sender":"service","connection_id":"","message":"initializing SFTP server with config {Banner:Private Server BindPort:2022 BindAddress: IdleTimeout:15 MaxAuthTries:5 Umask:0022 UploadMode:1 Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} Keys:[] IsSCPEnabled:false KexAlgorithms:[] Ciphers:[] MACs:[] LoginBannerFile: SetstatMode:0 EnabledSSHCommands:[md5sum sha1sum cd pwd]}"}
{"level":"debug","time":"2020-03-01T22:18:11.306","sender":"utils","connection_id":"","message":"umask not available on windows, configured value 0022 (18)"}
{"level":"info","time":"2020-03-01T22:18:11.306","sender":"sftpd","connection_id":"","message":"No host keys configured and \"C:\\\\ProgramData\\\\SFTPGO\\\\id_rsa\" does not exist; creating new private key for server"}

It just keeps repeating those lines over and over when the service restarts itself. It would help if it wrote better error messages in the sftpgo log file.

This unhelpful error message appears in the Windows Event Log called "System".

The SFTPGo service terminated with the following service-specific error: 
Incorrect function.

It would be nice if that Event Log error was also more helpful. Not sure if you have any control over that though, unless you can catch the exception and rethrow it with a more helpful "message" property.

Feature suggest: Let users be disabled and expire

Hi

It would be great to have users to be able to be disabled and expire.

Disabled:
Users in UNIX can be locked. This option doesn't seems to be implemented here.

Expire:
If we can ulitize the actions and http notification webhook to tell the datastore when a user was last used, we can utilise the API to disable/lock user after a certain time, if no last used variable was to be implemeted.

In UNIX we can use the lastlogin command to see when a user has last had it's successful connection.

Rest API via Postman or Curl

Hi,

I used the python rest client and was successful in creating users.

Is it possible to use the rest api via curl or postman?

I tried calling curl http://localhost:8080/add_user?username=test_username&password=test_pwd&home_dir=/tmp/test_home_dir&uid=33&gid=1000&max_sessions=2&quota_size=0&quota_files=3&permissions='*'&upload_bandwidth=100&download_bandwidth=60

but got an error {"error":"","message":"Not Found","status":404}

Thanks,

Ajay

Feature request: S3 Backend

It would be very convenient to have the ability to use the credentials: accesskey & secretkey of an s3 storage and then connect to the storage itself as backend.

Feature suggest: Public key in file

Hi

When giving the user a uid/guid it would also be nice to have the option to use a file as public key repository.

Just like normal users have it.

Restricting SFTP/SCP algorithms for security reasons

Hello,

Very great project.
Is it possible to add feature of restricting the use of SFTP/SCP Algorithms such as Compression algorithms, Key exchanges, ciphers, or MACs for security reasons ?
A white list to choose algorithms

Thanks