crazy-max/ftpgrab

Grab your files periodically from a remote FTP or SFTP server easily

About

FTPGrab is a CLI application written in Go and delivered as a single executable (and a Docker image) to grab your files from a remote FTP or SFTP server to your NAS, server or computer.

Note

Want to be notified of new releases? Check out 🔔 Diun (Docker Image Update Notifier) project!

Documentation

Documentation can be found on https://crazymax.dev/ftpgrab/

Contributing

Want to contribute? Awesome! The most basic way to show your support is to star the project, or to raise issues. You can also support this project by becoming a sponsor on GitHub or by making a PayPal donation to ensure this journey continues indefinitely!

Thanks again for your support, it is much appreciated! 🙏

License

MIT. See LICENSE for more details.
Icon credit to Nick Roach.

ftpgrab's Issues

Rewrite from scratch with Go

• Multiplatform : MacOS, FreeBSD, Linux, Windows, ARM, Android, Synology, QNAP
• Modern CLI interactions with cobra
• JSON Configuration file
• Detect and merge configuration
• Add Goreleaser
• Bolt db to store hash
• Native FTP client
• Logging with zerolog
• Send log through email
• Generate responsive emails through hermes
• Review Docker image
• Embedded cron using go routines
• Manage base dir
• Set original modtime
• Include/exclude based on regexp
• #39
• Handle mutex
• #9
• WebUI to customize config (maybe on a further release)

support sftp

• FTPGrab version (5.0.0) :
• Platform (freebsd) :

It would be nice if you could support sftp (ssh ftp). It looks like you might have in the past but it was removed? I would really like to use this, but would rather not run a ftp server when i already have ssh enabled.

Multiple errors trying to run on basic

Trying to run v2.03 on Transcend WiFi SD Linux fail with the following errors :

bash -x /etc/init.d/ftp-sync .

: not found/ftp-sync: line 10:
+
: not found/ftp-sync: line 44:

• CONFIG_FILE=/etc/ftp-sync/ftp-sync.conf
  +
  : not found/ftp-sync: line 46:
  +
  : not found/ftp-sync: line 48:
  +
  : not found/ftp-sync: line 50:
  /etc/init.d/ftp-sync: line 51: syntax error: unexpected "("

Make an installer

Add tests

Add ability to sync upload to FTP

I am currently using ftp-sync to backup the content of one FTP server to a local folder.
It would be great if we could use it to backup a local folder to a remote FTP server.

I have not yet digged deeper into the script, but I am hoping that replacing wget by wput would allow to do that easily.

What do you think?

Use type instead of which

Parallel downloads

Behaviour

Steps to reproduce this issue

1. Make config file with proper credentials
2. Start FTP Grabber
3. FTP Grabber starts to download, but single file simultaneously

Expected behaviour

I expected to see multiple downloads when I set connection per host.

Actual behaviour

It only downloads files sequentially, one by one.

Configuration

• FTPGrab version : 5.1.1
• Platform : windows
• Remote FTP : Filezilla-Server-windows

Logs

FTPGrab logs (set --log-level flag to debug and --log-ftp)

�[2mThu, 28 Mar 2019 10:09:00 UTC�[0m �[32mINF�[0m Starting FTPGrab 5.1.1
�[2mThu, 28 Mar 2019 10:09:00 UTC�[0m �[33mDBG�[0m {
  "Flags": {
    "Cfgfile": ".\\config_niko.yml",
    "Schedule": "",
    "Timezone": "UTC",
    "LogLevel": "debug",
    "LogJson": false,
    "LogFile": "",
    "Docker": false
  },
  "App": {
    "ID": "ftpgrab",
    "Name": "FTPGrab",
    "Desc": "Grab your files periodically from a remote FTP or SFTP server easily",
    "URL": "https://ftpgrab.github.io",
    "Author": "CrazyMax",
    "Version": "5.1.1"
  },
  "Server": {
    "Type": "ftp",
    "FTP": {
      "Host": "95.216.93.225",
      "Port": 21,
      "Username": "********",
      "Password": "********",
      "Sources": [
        "/Apex"
      ],
      "ConnectionsPerHost": 10,
      "Timeout": 5,
      "DisableEPSV": false,
      "TLS": {
        "Enable": false,
        "Implicit": true,
        "InsecureSkipVerify": false
      },
      "LogTrace": false
    },
    "SFTP": {
      "Host": "",
      "Port": 22,
      "Username": "********",
      "Password": "********",
      "Key": "********",
      "Sources": null,
      "Timeout": 30,
      "MaxPacketSize": 32768
    }
  },
  "Db": {
    "Enable": true,
    "Path": "apex_updater.db"
  },
  "Download": {
    "Output": "/Apex",
    "UID": 0,
    "GID": 0,
    "ChmodFile": 420,
    "ChmodDir": 493,
    "Include": null,
    "Exclude": null,
    "Since": "0001-01-01T00:00:00Z",
    "Retry": 3,
    "HideSkipped": false,
    "CreateBasedir": false
  },
  "Mail": {
    "Enable": false,
    "Host": "localhost",
    "Port": 25,
    "SSL": false,
    "InsecureSkipVerify": false,
    "Username": "********",
    "Password": "********",
    "From": "",
    "To": ""
  },
  "File": {
    "FileAttributes": 32,
    "CreationTime": {
      "LowDateTime": 2846841088,
      "HighDateTime": 30729172
    },
    "LastAccessTime": {
      "LowDateTime": 2846841088,
      "HighDateTime": 30729172
    },
    "LastWriteTime": {
      "LowDateTime": 45055992,
      "HighDateTime": 30729548
    },
    "FileSizeHigh": 0,
    "FileSizeLow": 532,
    "Reserved0": 0
  }
}
�[2mThu, 28 Mar 2019 10:09:00 UTC�[0m �[32mINF�[0m ########
�[2mThu, 28 Mar 2019 10:09:01 UTC�[0m �[33mDBG�[0m 0 entries found in database
�[2mThu, 28 Mar 2019 10:09:01 UTC�[0m �[32mINF�[0m ########
�[2mThu, 28 Mar 2019 10:09:01 UTC�[0m �[32mINF�[0m Grabbing from /Apex
�[2mThu, 28 Mar 2019 10:09:01 UTC�[0m �[32mINF�[0m --------
�[2mThu, 28 Mar 2019 10:09:01 UTC�[0m �[32mINF�[0m Checking /Apex/amd_ags_x64.dll
�[2mThu, 28 Mar 2019 10:09:01 UTC�[0m �[32mINF�[0m Already downloaded
�[2mThu, 28 Mar 2019 10:09:01 UTC�[0m �[31mWRN�[0m Skipped: Already downloaded
�[2mThu, 28 Mar 2019 10:09:02 UTC�[0m �[32mINF�[0m --------
�[2mThu, 28 Mar 2019 10:09:02 UTC�[0m �[32mINF�[0m Checking /Apex/audio/ship/audio.mprj
�[2mThu, 28 Mar 2019 10:09:02 UTC�[0m �[32mINF�[0m Already downloaded
�[2mThu, 28 Mar 2019 10:09:02 UTC�[0m �[31mWRN�[0m Skipped: Already downloaded
�[2mThu, 28 Mar 2019 10:09:02 UTC�[0m �[32mINF�[0m --------
�[2mThu, 28 Mar 2019 10:09:02 UTC�[0m �[32mINF�[0m Checking /Apex/audio/ship/general.mbnk
�[2mThu, 28 Mar 2019 10:09:02 UTC�[0m �[32mINF�[0m Exists but size is different
�[2mThu, 28 Mar 2019 10:09:02 UTC�[0m �[32mINF�[0m Downloading file (56.03MB) to /Apex/audio/ship/general.mbnk...
�[2mThu, 28 Mar 2019 10:16:32 UTC�[0m �[32mINF�[0m File successfully downloaded!
�[2mThu, 28 Mar 2019 10:16:32 UTC�[0m �[32mINF�[0m Time spent: 7 minutes
�[2mThu, 28 Mar 2019 10:16:32 UTC�[0m �[32mINF�[0m --------
�[2mThu, 28 Mar 2019 10:16:32 UTC�[0m �[32mINF�[0m Checking /Apex/audio/ship/general.mbnk_digest
�[2mThu, 28 Mar 2019 10:16:32 UTC�[0m �[32mINF�[0m Never downloaded
�[2mThu, 28 Mar 2019 10:16:32 UTC�[0m �[32mINF�[0m Downloading file (1.82MB) to /Apex/audio/ship/general.mbnk_digest...
�[2mThu, 28 Mar 2019 10:16:47 UTC�[0m �[32mINF�[0m File successfully downloaded!
�[2mThu, 28 Mar 2019 10:16:47 UTC�[0m �[32mINF�[0m Time spent: 14 seconds
�[2mThu, 28 Mar 2019 10:16:47 UTC�[0m �[32mINF�[0m --------
�[2mThu, 28 Mar 2019 10:16:47 UTC�[0m �[32mINF�[0m Checking /Apex/audio/ship/general_english.mstr
�[2mThu, 28 Mar 2019 10:16:47 UTC�[0m �[32mINF�[0m Never downloaded
�[2mThu, 28 Mar 2019 10:16:47 UTC�[0m �[32mINF�[0m Downloading file (848.7MB) to /Apex/audio/ship/general_english.mstr...

Add multiple ftp sources paths

Add wiki documentation

Bug download with sqlite3

Connection Per Host is not working

Dears,
First of all thanks for developing such a simple and reliable tool. Second, I want to know why the connection_per_host parameter is not working at all?
I've set the parameter to 10, but it only transfer 1 file simultaneously. I want to make it faster with multiple connections, what should i do now ?

ftp:
  host: 127.0.0.1
  port: 21
  username: test
  password:
  connections_per_host: 10
  timeout: 5
  disable_epsv: false
  tls:
    enable: false
    implicit: true
    insecure_skip_verify: false
  sources:
    - /Src

db:
  enable: true
  path: update.db

download:
  output: Dest\
  uid:
  gid:
  include:
  exclude:
  since: 0001-01-01T00:00:00Z
  retry: 3
  hide_skipped: false
  create_basedir: false

Bug trailing slash

--------------
Process file : ...
Hash: 2ed0b237f11c06740893638275d52996
Size: 
Status : Never downloaded...
Start download to ...... Please wait...
ERROR 82: Download failed... retry 1/3

hash txt error

Hi,

ftpgrab is nice software. I use it with hash storage text. I have got error

/usr/local/ftpgrab/ftpgrab.sh: line 73: [: 76cb7ce2867a671c86bd35f0697a902b: binary operator expected
Process file: 24techno/EPG-12-01-2018.xml
Hash: 76cb7ce2867a671c86bd35f0697a902b
Size: 5.34 Kb
Status: Already downloaded and valid. Skip download...
Time spent: 00:00:01

Cannot read directory error if filename contains ; char

If a file in a directory (FTP server) contains a ; the following error occurred :

Thu, 18 Jul 2019 19:32:18 UTC ERR Cannot read directory /test/ error="failed parsing MLST entry: type=file;size=7903718;modify=20190718185113;UNIX.mode=0644;UNIX.uid=1001;UNIX.gid=1001;unique=903g5b00215; file;test.txt" source=/test/

Bug related to secsy/goftp module.

Rename the project ftpgrab !

As this project is a one way sync it has to be renamed

unable to use config

Behaviour

I have copied the sample yml files and get numerous errors when trying to use them. I'm assuming I must be doing something wrong here.

Steps to reproduce this issue

1.create config
2.run
3.errors

Expected behaviour

hopefully work lol

Actual behaviour

Errors like this:

./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:15:15 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:15:15 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 5: mapping values are not allowed in this context"

if I delete the space before the number on line 5 it will run until it hits another space... then I get other weird errors. I'm using notepad++ to create and edit the yml

Configuration

• FTPGrab version (5.0.0) : 5.3.0
• Platform (windows/linux/freebsd/macos) : Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-51-generic x86_64)
• Remote FTP (Pure-FTPd) or SFTP (OpenSSH) server: not getting that far, tried both

server:
type: ftp
ftp:
host:IPADDRESS
port:22222
username:fake
password:password
sources:/rtorrent/download
- /
timeout:5 #I'm having to remove all these spaces or I get errors.
disable_epsv:false
tls:
enable: true
implicit: true
insecure_skip_verify: false
log_trace: false

db:
enable: true
path: ftpgrab.db

download:
output: /rtorrent/download
uid:
gid:
chmod_file: 0644
chmod_dir: 0755
include:
exclude:
since: 0001-01-01T00:00:00Z
retry: 3
hide_skipped: false
create_basedir: false

Logs

FTPGrab logs (set --log-level flag to debug and --log-ftp)

Tue, 04 Jun 2019 16:29:16 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 16:29:16 UTC FTL Cannot load configuration error="unable to open config file, lstat ftpgrab.yml: no such file or directory"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 16:29:39 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 16:29:39 UTC FTL Cannot load configuration error="unable to open config file, lstat ./ftpgrab.yml: no such file or directory"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 16:30:44 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 16:30:44 UTC FTL Improper configuration error="FTP host is required"
fake@fake:~$ ftpgrab --config ftpgrab.yml
-bash: ftpgrab: command not found
fake@fake:~$ ./ftpgrab --config ftpgrab.yml
Tue, 04 Jun 2019 16:31:05 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 16:31:05 UTC FTL Improper configuration error="FTP host is required"
fake@fake:~$ ./ftpgrab --help
usage: ftpgrab --config=CONFIG [<flags>]

Grab your files periodically from a remote FTP or SFTP server easily. More info
on https://ftpgrab.github.io

Flags:
  --help               Show context-sensitive help (also try --help-long and
                       --help-man).
  --config=CONFIG      FTPGrab configuration file.
  --schedule=SCHEDULE  CRON expression format.
  --timezone="UTC"     Timezone assigned to FTPGrab.
  --log-level="info"   Set log level.
  --log-json           Enable JSON logging output.
  --log-file=LOG-FILE  Add logging to a specific file.
  --docker             Enable Docker mode.
  --version            Show application version.

fake@fake:~$ ./ftpgrab --config=config
Tue, 04 Jun 2019 16:31:49 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 16:31:49 UTC FTL Cannot load configuration error="unable to open config file, lstat config: no such file or directory"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 16:32:08 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 16:32:08 UTC FTL Improper configuration error="FTP host is required"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:15:15 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:15:15 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 5: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config
ftpgrab: error: expected argument for flag '--config', try --help
fake@fake:~$ ./ftpgrab --config --help
ftpgrab: error: expected argument for flag '--config', try --help
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
[sudo] password for sboob:
Tue, 04 Jun 2019 17:15:58 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:15:58 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 5: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config
ftpgrab: error: expected argument for flag '--config', try --help
fake@fake:~$ ./ftpgrab --config ftpgrab.yml
Tue, 04 Jun 2019 17:16:35 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:16:36 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 5: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:17:17 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:17:17 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 5: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:17:39 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:17:39 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 5: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:17:55 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:17:55 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 5: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:17:57 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:17:57 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 5: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:18:17 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:18:17 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 8: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:18:38 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:18:38 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 8: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:18:50 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:18:50 UTC FTL Improper configuration error="FTP host is required"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:20:25 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:20:25 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 5: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:20:38 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:20:38 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 10: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:21:45 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:21:45 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:22:03 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:22:06 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:22:26 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:22:26 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:22:47 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:22:48 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:23:12 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:23:12 UTC FTL Improper configuration error="FTP host is required"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 17:23:39 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:23:39 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 5: mapping values are not allowed in this context"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 17:23:49 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:23:49 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 10: mapping values are not allowed in this context"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 17:23:56 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:23:57 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 11: mapping values are not allowed in this context"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 17:24:06 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:24:06 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 17:31:53 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:31:53 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 8: mapping values are not allowed in this context"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 17:40:14 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:40:14 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 10: mapping values are not allowed in this context"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 17:40:27 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:40:27 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 11: mapping values are not allowed in this context"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 17:40:35 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:40:35 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 17:40:49 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:40:49 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 15: did not find expected key"
fake@fake:~$ sudo ./ftpgrab --config ./ftpgrab.yml
Tue, 04 Jun 2019 17:41:03 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:41:03 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 15: did not find expected key"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:41:26 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:41:26 UTC FTL Improper configuration error="FTP host is required"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:41:33 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:41:35 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:41:40 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:41:40 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:43:02 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:43:02 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:43:23 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:43:23 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:43:34 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:43:34 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:45:20 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:45:20 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 8: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:45:35 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:45:35 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 8: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:45:45 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:45:45 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 12: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:45:54 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:45:55 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: line 13: mapping values are not allowed in this context"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:46:08 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:46:08 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: unmarshal errors:\n  line 4: cannot unmarshal !!str `host:88...` into model.FTP"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:46:36 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:46:36 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: unmarshal errors:\n  line 4: cannot unmarshal !!str `host:ft...` into model.FTP"
fake@fake:~$ ./ftpgrab --config=ftpgrab.yml
Tue, 04 Jun 2019 17:46:57 UTC INF Starting FTPGrab 5.3.0
Tue, 04 Jun 2019 17:46:57 UTC FTL Cannot load configuration error="unable to decode into struct, yaml: unmarshal errors:\n  line 4: cannot unmarshal !!str `host:/8...` into model.FTP"

#at this point it almost seems to be ready to work but can't figure out that I put an IP there?

wget error 4

Sorry of this is the wrong place to ask but I cannot find anywhere else!

I have just installed ftpgrab but cannot get it to work?

If I try a manual wget command to download a file it works fine but I cannot get ftpgrab to work due to a wget error 4, can someone please look over my log to see if there is anything obviously wrong?

I have *** out my username and password purposely

Many Thanks!

FTPGrab v4.1.1 (rapidseedbox - 2017/10/11 01:09:12

Config: rapidseedbox
Script PID: 5508
Log file: /var/log/ftpgrab/rapidseedbox-20171011010912.log
FTP sources count: 1
FTP secure: 0
Download method: wget
Resume downloads: 0
Shuffle file/folder list: 0
Hash type: md5
Hash storage: text
Hash file: /opt/ftpgrab/hash/rapidseedbox.txt

#DEBUG FTP_SRC: /Downloads/
#DEBUG DIR_DEST: /home/ubuntuvm/unraidserver/processingfolder/ubuntuvmsync
#DEBUG DIR_DEST_REF: /home/ubuntuvm/unraidserver/processingfolder/ubuntuvmsync/
Source: ftp://.../Downloads/
Destination: /home/ubuntuvm/unraidserver/processingfolder/ubuntuvmsync/
Checking connection to ftp://.../Downloads/...
#DEBUG checkConnection: wget --spider -q --tries=1 --timeout=5 --ftp-user=***** --ftp-password=********** -O - ftp://.../Downloads/
ERROR: Wget error 4
More infos: http://www.gnu.org/software/wget/manual/html_node/Exit-Status.html

Allow multiple instances

Would be nice to be able to have configurations for more than one FTP host or more than one FTP user on the same host. Maybe simply add a parameter to specify the conf file, defaults to ftp-sync.conf

Sed not escaping & char

Remove local files that no longer exist on the source server

It would be a nice feature to remove the local files that no longer exist on the source server. The actual implementation does not remove these files and the local copy of the source server grows indefinitely.. This removal feature could be turned on/off by a property in the config file.

Ignore files by date?

Hi,

I was wondering if there was a way to set ftpgrab to only look at files in a directory, starting from a particular date?

I'd like to set it up to monitor a remote dir, but don't need it pulling in a bunch of files that exist remotely that I have already downloaded and sorted.

Thanks for the help.

Syntax error

each time I run this script on my debian wheezy I get the following error
/etc/init.d/ftp-sync: 74: /etc/init.d/ftp-sync: Syntax error: "(" unexpected

I have no idea what i could have done wrong...

line 73 and line 88 [: too many arguments

I was getting an error while executing the script (/usr/bin/ftpgrab: line 73: [: too many arguments), but I found the solution:

in /usr/bin/ftpgrab,
line 73:
change:
if [ "$HASH_STORAGE" == "text" ] && [ -z $(grep "^$_SRC_HASH" "$HASH_FILE") ]
to:
if [ "$HASH_STORAGE" == "text" ] && [[ -z $(grep "^$_SRC_HASH" "$HASH_FILE") ]]

line 88:
change:
if [ "$HASH_STORAGE" == "text" ] && [ ! -z $(grep "^$_SRC_HASH" "$HASH_FILE") ]
to:
if [ "$HASH_STORAGE" == "text" ] && [[ ! -z $(grep "^$_SRC_HASH" "$HASH_FILE") ]]

Maybe you could do it directly in your next release :)

Always getting Status: Hash sum exists. Skip download...

I seem to get this result when trying to download a file with the same name as one previously downloaded. However I know that the file has changed, and the sha1sum does not match what is in the hash db. The output even indicates the file size is different, but displays an incorrect sha1sum than if I download the file and run sha1sum filename

Création de dossiers supplémentaires

Bonsoir,

Comme demander voici le ticket ;)

Ma version OS : BusyBox v1.16.1 - Noyau 2.6.32.12

Mon problème : pour chaque création de dossier lors des synchronisations, ça ajoute la destination que l'on passe en paramètre.

Extrait de log sachant que ma destination est /volume1/seedbox :

Process file : 9.Mois.Ferme.2013.FRENCH.720p.BluRay.x264-LOSTlost-9moisferme.720p.jpg
Hash: 685a3da54bc5eea1d119eb6f9d896477
Size: 15.70 Kb
Status : Never downloaded...
Start download to /volume1/seedbox/9.Mois.Ferme.2013.FRENCH.720p.BluRay.x264-LOST/volume1/seedbox/lost-9moisferme.720p.jpg... Please wait...
File successfully downloaded!

Time spent: 00:00:02

Skip download when path contain character like "[" and "]"

Hi :)

Behaviour

Downloads are not started when parent folder name contain brackets as a character [ or ]

Steps to reproduce this issue

1. create folder with brackets caracter like [ or ], for example directory.[team]/filename.mkv
2. grabftp list the file, send the command, but do not start download
3. configure disable_epsv: true

Expected behaviour

Tue, 30 Apr 2019 09:30:21 UTC INF Never downloaded
Tue, 30 Apr 2019 09:30:21 UTC INF Downloading file (993.4MB) to /path/filename...
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.209 #1 was ready
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.209 #1 sending command TYPE I
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.232 #1 got 200-TYPE is now 8-bit binary
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.232 #1 sending command SIZE /path/filename
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.256 #1 got 213-993427745
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.257 #1 was ready
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.257 #1 was ready
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.257 #1 sending command TYPE I
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.280 #1 got 200-TYPE is now 8-bit binary
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.280 #1 sending command PASV
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.304 #1 got 227-Entering Passive Mode (@ip)
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.305 #1 opening data connection to @IP
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.328 #1 upgrading data connection to TLS
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.329 #1 sending command RETR /path/filename
Tue, 30 Apr 2019 09:30:21 UTC DBG goftp: 3.351 #1 got 150-Accepted data connection
970144.3 kbytes to download

Actual behaviour

Tue, 30 Apr 2019 09:18:52 UTC DBG goftp: 1.237 #1 opening data connection to @ip
Tue, 30 Apr 2019 09:18:52 UTC DBG goftp: 1.263 #1 upgrading data connection to TLS
Tue, 30 Apr 2019 09:18:52 UTC DBG goftp: 1.263 #1 sending command MLSD /path[team]/filename
Tue, 30 Apr 2019 09:18:52 UTC DBG goftp: 1.287 #1 got 150-Accepted data connection
Tue, 30 Apr 2019 09:18:52 UTC DBG goftp: 1.337 #1 was ready
Tue, 30 Apr 2019 09:18:52 UTC DBG goftp: 1.337 #1 sending command PASV
Tue, 30 Apr 2019 09:18:52 UTC DBG goftp: 1.360 #1 got 227-Entering Passive Mode (@ip)
Tue, 30 Apr 2019 09:18:52 UTC DBG goftp: 1.360 #1 opening data connection to @IP
Tue, 30 Apr 2019 09:18:52 UTC DBG goftp: 1.382 #1 upgrading data connection to TLS
Tue, 30 Apr 2019 09:18:52 UTC DBG goftp: 1.382 #1 sending command MLSD /path[team]/filename
Tue, 30 Apr 2019 09:18:52 UTC DBG goftp: 1.404 #1 got 150-Accepted data connection

no download are started

Configuration

• FTPGrab version : 5.2.0
• Platform : linux mint
• Remote server: Pure-FTPd

Logs

in step before

configuration :

server:
  type: ftp
  ftp:
    host: xxxxxxxxxx
    port: 21
    username: XXXXXX
    password: XXXXXX
    sources:
      - /path
    timeout: 5
    disable_epsv: true
    tls:
      enable: yes
      implicit: false
      insecure_skip_verify: true
    log_trace: true

db:
  enable: true
  path: ftpgrab.db

download:
  output: /path
  uid:
  gid:
  chmod_file: 0755
  chmod_dir: 0755
  since:
  retry: 3
  hide_skipped: false
  create_basedir: false

Thanks, :)

New feature suggestions

ftpgrab is a great software, but in the process of using, I hope to add the following features：

1. finished check，detecting whether the file has been written，such as delay detection file size（First check the file size once, and then check it again after 10 seconds. If the two sizes are equal, the file is completed.），suffix filtering（just like sample.txt and sample.txt.writting），ok file check（There is an sample.ok file indicating that the sample.txt file is completed. ）etc.
2. path Expression support，Expressions are allowed in the path，just like /data/{yyyyMMdd}/ or Regular expression like /data//^\d{8}^//
3. execute command after file transfer completed, and support some built-in parameters ，like file path ,size etc....

The server files updated not sync to local

The server files updated not sync to local, the sample file name is not sync.

Can you support this feature? eg: check sum for content ,not file name.

ftpgrab trying to download unfinished downloads

Hi,

I love this little program, it works very very well for me except I am getting failures due to ftpgrab trying to download a file from my ftp that has not finished downloading.. is there a way around this?

Many Thanks!

Checking file download integrity

Thank you for your post, I think this may bring an answer to my question: I am trying to implement this routine:

1-Copy files from FTP server to my local machine.
2-Check if the downloaded files are fine (i.e. md5checksum).
3-If the download was fine then delete the downloaded files from the FTP server.

And I shoud have an independant file that contains the association between the file located at the FTP server and its MD5. Then I should be able to compare locally if the downloaded file is fine or was corrupted.

-Does your script adress this senario ?
-On which field are you running the MD5 checksum ?
-I need to remove the file that was downloaded from the FTP server while having it locally. Does your script allows me to do that ?

Thank you so much.

One time

Hello
It is possible that a file could be download juste one time? I want to be able to delete the destination file without the script download again. Or it is possible to delete the source when the file is download?

Regards

Can't save downloaded files

Hi,
this is a cool script. Thanks for that. The issue I get is:

"Start download to /home/raton/Dokumente/daily_station_data/bash_completion_tmux.sh... Please wait..."

• wget --progress=dot:mega --ftp-user=XXXXXX --ftp-password=XXXXXX -O /home/raton/Dokumente/daily_station_data/bash_completion_tmux.sh ftp://ftp.XXXXX.com:21/suntrace/daily_station_data/bash_completion_tmux.sh
• ftpsyncProgressFilter
• '[' 1 == 0 ']'
• local errordl=0

and

• ftpsyncEcho 'ERROR 02: Download failed... retry 1/3'
• echo -e 'ERROR 02: Download failed... retry 1/3'
• tee -a /usr/bin/ftp-sync/logs/ftp-sync-20130924162052.log
  ERROR 02: Download failed... retry 1/3
• ftpsyncDownloadFile /suntrace/daily_station_data/bash_completion_tmux.sh /home/raton/Dokumente/daily_station_data/bash_completion_tmux.sh '' 1

I checked folder permissions and run the script as root. Same behaviour.
I changed the DL_USER and DL_CHMOD. Same. Maybe you can get me an approach. Have a nice one!

error with awk

Thanks for your reply !

Now i don't have the previous syntax error and the script can connect to the distant ftp server without any problem.

It can also get the list of files to download but each time it fails and i have these erros :

ERROR 02: Download failed and too many retries...

Time spent: 00:00:05

awk: line 1: syntax error at or near *
awk: line 1: syntax error at or near *
awk: line 1: syntax error at or near *
awk: line 1: extra ')'
awk: line 1: syntax error at or near {

Still running it on a debian 7 server...

Files downloaded again (with hash already in database)

Hi !

For several days, some files that have already been downloaded are downloaded again (with the same hash). I noticed that on small files (sfv, nfo), the problem doesn't happen.
If you have any idea what the cause of the problem is, I would be grateful:)
I dont understand, because ftpgrab work fine since few weeks and since few day it doesnt work properly :s
I try to upgrade from 4.1 to 4.2, it doesn't change anything. I try to remove all ftpgrab files (db, config) and reinstall it.
I try with file storage for hash.

Steps to reproduce this issue

• Start a download sync with ftpgrab
• remove downloaded files from destination but keep files from source(ftp)
• re-sync with ftpgrap
• The files already downloaded before are downloaded again (same hash)...

Expected Behaviour

it should not download again the files with the same hash

Actual Behaviour

it download file with when the hash is preent in the sqlite database

Configuration

Operating system and platform (ex.Debian 8 64bits) :
Mint 64bits

ftpgrab version (ex. 3.0) :
4.1 and the same error after upgrading to 4.2

wget version (ex. 1.16) :
GNU Wget 1.17.1 compilé sur linux-gnu

curl version (ex. 7.38.0) :
7.47.0

md5sum / sha1sum version (ex. 8.23) :
sha1sum (GNU coreutils) 8.25

sqlite3 version (ex. 3.8.7.1) :
3.11.0 2016-02-15

---

Some logs with a file was already in database and already downloaded :

FTPGrab v4.2.1 (**** - 2017/10/29 05:00:02)

Config: ****
Script PID: 5351
Log file: /var/log/ftpgrab/-20171029050002.log
FTP sources count: 1
FTP secure: 1
Download method: curl
Resume downloads: 0
Shuffle file/folder list: 0
Hash type: sha1
Hash storage: sqlite3
Hash file: /opt/ftpgrab/hash/.db

#DEBUG FTP_SRC: //TERMINE//DOWNLOAD/
#DEBUG DIR_DEST: /media//*
#DEBUG DIR_DEST_REF: /media///
Source: ftp://..com://TERMINE//DOWNLOAD/
Destination: /media///
Checking connection to ftp://..://TERMINE//DOWNLOAD/...
#DEBUG checkConnection: curl --silent --retry 1 --retry-delay 5 --globoff -u : --ftp-ssl --insecure ftp://..com://TERMINE//DOWNLOAD/
Successfully connected!
#DEBUG checkfolder: curl --silent --list-only --globoff -u : --ftp-ssl --insecure "ftp://..com://TERMINE//DOWNLOAD/%5f1080/"
#DEBUG lineClean: _1080/
#DEBUG basename: _1080
#DEBUG srcfile: //TERMINE//DOWNLOAD/_1080
#DEBUG srcfileproc: //TERMINE//DOWNLOAD/%5f1080
#DEBUG srcfileshort: _1080
#DEBUG srcfileshort2: _1080
#DEBUG srchash: 778870d88b238ebea917bbd03af402b8a0d11fe5
#DEBUG srcsize:
#DEBUG destfile: /media///_1080
#DEBUG destsize: N/A
#DEBUG vregex: _1080
#DEBUG checkfolder: curl --silent --list-only --globoff -u : --ftp-ssl --insecure "ftp://..com://TERMINE//DOWNLOAD/%5f1080/%5b%20********%20%5d%20**.mkv/"
#DEBUG lineClean: [ ******* ] .mkv
#DEBUG basename: [ ******** ] .mkv
#DEBUG srcfile: //TERMINE//DOWNLOAD/_1080/[ ******** ] .mkv
#DEBUG srcfileproc: //TERMINE//DOWNLOAD/%5f1080/%5b%20********%20%5d%20*.mkv
#DEBUG srcfileshort: _1080/[ ******** ] .mkv
#DEBUG srcfileshort2: _1080/[ ******** ] .mkv
#DEBUG srchash: aa5c5b6f229f96719be96073bf11ad850b0a4ed9
#DEBUG srcsize: 8526998229
#DEBUG destfile: /media///_1080/[ ******** ] .mkv
#DEBUG destsize: N/A
#DEBUG vregex: _1080/[ ******** ] .mkv
Process file: _1080/[ ******** ] T.mkv
Hash: aa5c5b6f229f96719be96073bf11ad850b0a4ed9
Size: 7.94 Gb
Status: Never downloaded...
Start download to /media///_1080/[ ******** ] .mkv... Please wait...
#DEBUG Download command: curl --globoff -u : --ftp-ssl --insecure "ftp://..com://TERMINE//DOWNLOAD/%5f1080/%5b%20********%20%5d%20**.mkv" -o "/media///_1080/[ ******** ] ***.mkv"

ftpgrab restarts automatically after being killed

Hi,

I can't get ftpgrab to stop after killing it.
The process always starts again
Thanks

Synology Package

Is there a synology package for this somewhere?

Thanks

Ref: http://forum.synology.com/wiki/index.php/Synology_package_files

Continue downloads

First of all, thanks for the script!
I love the idea if storing md5 of downloaded files.
What about continuable downloads in case of large files/slow connection?
I tried the option "-C -" for curl (wget has a similar option) and it seems to work fine after restarting the script.
Is there a reason you didn't include this option?

Syntax error

Hi,
I encountered several syntax error with my Synology NAS... The OS version is DSM 5 beta

ftp-sync: line 51: syntax error: unexpected "("

After removed "function " from function name, it is working, but encounter another error, it is:

ftp-sync: line 176: syntax error: unexpected redirection

ftp-sync: line 294: syntax error: bad substitution

Thank you.

Use SQLite for MD5 hash

Use a SQLite database for MD5 storage.

host is required but specified

ftpgrab freebsd 64bit binary from the 5.1.0 release. Two issues.

First I get this even though host is defined in sftp.

Fri, 15 Feb 2019 00:23:38 UTC FTL Improper configuration error="host is required"

ftpgrab --version reprots "dev" instead of the actual version number.

nawk on DS216j / DSM 6.1.6-15266 Update 1

Hi,

nawk is not available on DSM 6.1.6-15266 Update 1.
I installed entware but its not there too.
I commented the nawk part on the script and it seems to be fine though.
Thanks for your great script anyway :)

Need to run as root/sudo

When running as the local user, the script fails to create /var/run/ftp-sync.pid and the log files associated with the sync.

I'm required to run the script as a sudo user.

Any workarounds?

Bug checking MD5 (sqlite3 and text)

SFTP option

I don't think curl has a built in sftp option right? I think that's the only thing missing from this script! Awesome job but I won't use it until I can do encrypted / secure transfers. Could you add this option? Thanks!

Empty folder leeds to spinlock

Hi,
first of all thanks for this great script!

When trying to use it, my example directory structure contained an empty folder which lead to the script hanging forever.

With debug enabled I could see how it loops. The only thing that changes is the srchash
As soon as I added an empty file to this folder, the problem was gone.

Here is an excerpt of the debug output (I censored the real names)

#DEBUG checkfolder: curl --silent --list-only --globoff -u *****:***** --ftp-ssl "ftp://example.com:21/AN%5fEMPTY%5fFOLDER//////////////////////"
#DEBUG lineClean: /
#DEBUG basename: /
#DEBUG srcfile: /AN_EMPTY_FOLDER//////////////////////
#DEBUG srcfileproc: /AN%5fEMPTY%5fFOLDER//////////////////////
#DEBUG srcfileshort: AN_EMPTY_FOLDER//////////////////////
#DEBUG srcfileshort2: AN_EMPTY_FOLDER//////////////////////
#DEBUG srchash: 0e087c88145ec4a845f2f9d575560747
#DEBUG srcsize:
#DEBUG destfile: /tmp/seedbox/AN_EMPTY_FOLDER//////////////////////
#DEBUG destsize: N/A
#DEBUG vregex: AN_EMPTY_FOLDER//////////////////////
#DEBUG checkfolder: curl --silent --list-only --globoff -u *****:***** --ftp-ssl "ftp://example.com:21/AN%5fEMPTY%5fFOLDER////////////////////////"
#DEBUG lineClean: /
#DEBUG basename: /
#DEBUG srcfile: /AN_EMPTY_FOLDER////////////////////////
#DEBUG srcfileproc: /AN%5fEMPTY%5fFOLDER////////////////////////
#DEBUG srcfileshort: AN_EMPTY_FOLDER////////////////////////
#DEBUG srcfileshort2: AN_EMPTY_FOLDER////////////////////////
#DEBUG srchash: 123b24923553082f8beee2004ed55496
#DEBUG srcsize:
#DEBUG destfile: /tmp/seedbox/AN_EMPTY_FOLDER////////////////////////
#DEBUG destsize: N/A
#DEBUG vregex: AN_EMPTY_FOLDER////////////////////////
#DEBUG checkfolder: curl --silent --list-only --globoff -u *****:***** --ftp-ssl "ftp://example.com:21/AN%5fEMPTY%5fFOLDER//////////////////////////"
#DEBUG lineClean: /
#DEBUG basename: /
#DEBUG srcfile: /AN_EMPTY_FOLDER//////////////////////////
#DEBUG srcfileproc: /AN%5fEMPTY%5fFOLDER//////////////////////////
#DEBUG srcfileshort: AN_EMPTY_FOLDER//////////////////////////
#DEBUG srcfileshort2: AN_EMPTY_FOLDER//////////////////////////
#DEBUG srchash: af9471cd3ecc2715fdf5c276accbae98
#DEBUG srcsize:
#DEBUG destfile: /tmp/seedbox/AN_EMPTY_FOLDER//////////////////////////
#DEBUG destsize: N/A
#DEBUG vregex: AN_EMPTY_FOLDER//////////////////////////
#DEBUG checkfolder: curl --silent --list-only --globoff -u *****:***** --ftp-ssl "ftp://example.com:21/AN%5fEMPTY%5fFOLDER////////////////////////////"
#DEBUG lineClean: /
#DEBUG basename: /
#DEBUG srcfile: /AN_EMPTY_FOLDER////////////////////////////
#DEBUG srcfileproc: /AN%5fEMPTY%5fFOLDER////////////////////////////
#DEBUG srcfileshort: AN_EMPTY_FOLDER////////////////////////////
#DEBUG srcfileshort2: AN_EMPTY_FOLDER////////////////////////////
#DEBUG srchash: 4515adb96595251e22d416fd7c8bbbb9
#DEBUG srcsize:
#DEBUG destfile: /tmp/seedbox/AN_EMPTY_FOLDER////////////////////////////
#DEBUG destsize: N/A
#DEBUG vregex: AN_EMPTY_FOLDER////////////////////////////

How to exclude .part

Hi Crazy-max,

How can I exclude tha .part files?

Thanks for all

add ftpgrab train command

It would be nice if I could point ftpgrab to a directory of files and it would hash them so it wouldn't download them again. Sometimes it's not as simple as setting a date on from when to grab. I think this would be a nice way to train it before you use it so it's not downloading anything you already have.

directory name has "." bug

Behaviour

When ftp server has directory name contains ".", read direcotry files fail.

Steps to reproduce this issue

1. Create direcotry like "data-2019.1-12"
2. Upload a file to the diredtory

Expected behaviour

I can download the file is ok!

Actual behaviour

I skip the direcotry, and can't found file.

Configuration

• FTPGrab version (5.0.0) : 5.3.0
• Platform (windows/linux/freebsd/macos) : Linux
• Remote FTP (Pure-FTPd) or SFTP (OpenSSH) server: ftp

Problem

When i debug for it, i found ftp read dir that code:

// ReadDir fetches the contents of a directory, returning a list of os.FileInfo's
func (c *Client) ReadDir(path string) ([]os.FileInfo, error) {
	return c.ftp.ReadDir(regexp.QuoteMeta(path))
}

It's egexp.QuoteMeta function report the bug, it convert:

"data-2019.1-12" -> "data-2019\.1-12"

But the ftp server not exists folder "data-2019.1-12"!

When I remove "regexp.QuoteMeta", it run ok!

Is support config charset like GB2312?

Behaviour

A chinese ftp server, the folder like "中文目录", it can't create the correct directory.

Steps to reproduce this issue

1. A ftp server set charset GB2312
2. use ftpgrab

Configuration

• FTPGrab version (5.0.0) : 5.3.0
• Platform (windows/linux/freebsd/macos) : linux
• Remote FTP (Pure-FTPd) or SFTP (OpenSSH) server: security and private, sorry!

Logs

FTPGrab logs (set --log-level flag to debug and --log-ftp)

Mon, 03 Jun 2019 08:50:52 UTC INF --------
Mon, 03 Jun 2019 08:50:52 UTC INF Checking /2018������/����/����-EXP-201812.rar
Mon, 03 Jun 2019 08:50:52 UTC INF Never downloaded
Mon, 03 Jun 2019 08:50:52 UTC INF Downloading file (10.15MB) to /data/�/����-EXP-201812.rar...
Mon, 03 Jun 2019 08:50:52 UTC ERR Cannot create destination dir error="mkdir /data/2018������: invalid argument"
Mon, 03 Jun 2019 08:50:52 UTC INF Time spent: 3 milliseconds
Mon, 03 Jun 2019 08:50:53 UTC INF --------
Mon, 03 Jun 2019 08:50:53 UTC INF Checking /2019������/����͢/����͢-EXP-201901.rar
Mon, 03 Jun 2019 08:50:53 UTC INF Never downloaded
Mon, 03 Jun 2019 08:50:53 UTC INF Downloading file (10.3MB) to /data/2019������/����͢/����͢-EXP-201901.rar...
Mon, 03 Jun 2019 08:50:53 UTC ERR Cannot create destination dir error="mkdir /data/2019������: invalid argument"
Mon, 03 Jun 2019 08:50:53 UTC INF Time spent: 3 milliseconds
Mon, 03 Jun 2019 08:50:53 UTC INF --------
Mon, 03 Jun 2019 08:50:53 UTC INF Checking /2019������/����͢/����͢-EXP-201902.rar
Mon, 03 Jun 2019 08:50:53 UTC INF Never downloaded
Mon, 03 Jun 2019 08:50:53 UTC INF Downloading file (10.02MB) to /data/2019������/����͢/����͢-EXP-201902.rar...
Mon, 03 Jun 2019 08:50:53 UTC ERR Cannot create destination dir error="mkdir /data/2019������: invalid argument"
Mon, 03 Jun 2019 08:50:53 UTC INF Time spent: 3 milliseconds