willmorrison1/files-transfer

Files transfer

The script is made to send data using LFTP to an FTP server. The script use LFTP log file to determine the last file sent. This way, data are not constantly re-uploaded to the server.

The original script was made by Marc-Antoine https://github.com/madrouin https://gitlab.in2p3.fr/ipsl/sirta/dev-info/tools/files-transfer/-/blob/main/lftp_transfer.py

Requirements

• Code should work with python >=3.7
• Only dependency is toml package

Configuration of script

Create a config_transfer.toml file following example in conf directory

[FTP]
server = "ftp.server.fr"
user = "ftp_user"
password = "ftp_password"
port = 21
dir = "/ftp/directory"

[files]
dir_mask = "test/data"
file_mask = "T3250605_%Y%m%d_%H%M%S.nc"

The script will look at the options dir_mask and file_mask in the [files] section. You can use the format code below to indicate the format of dates and time in directory and file name

Directive	Meaning	Example
%Y	Year with century	0001, .., 2022
%y	Year without century (zero-padded)	00, .., 22
%m	Month (zero-padded)	01, .., 12
%d	Day (zero-padded)	01, .., 31
%H	Hour (zero-padded)	00, .., 23
%M	Minute (zero-padded)	00, .., 59
%S	Second (zero-padded)	00, .., 59

Running the script

First run

For the first run, the script needs to create the first lftp log file and so it needs to be provided the first date to look for files. e.g. here the first date to look for files is 2023-01-15

python lftp_transfer.py conf/conf_transfer.toml test/log/test.log --since '2023-01-15'

Other runs

python lftp_transfer.py conf/conf_transfer.toml test/log/test.log

files-transfer's Issues

add optional command line argument to replace `60` minutes: the age a file needs to be before transfer start.

files-transfer/lftp_transfer.py

Line 105 in 86be51e

return max(list_dates) - dt.timedelta(minutes=60)

if lftp has nothing to do, then there will be no files sent, and no storage of "--since" in the lftp log

If all files have been transferred since the last execution, then lftp will not send anything, and no files will be in the log file. Then there will be no last date in the log. Then --since will need to manually be set

files-transfer/lftp_transfer.py

Line 71 in 86be51e

def find_last_date_in_log(log_file: Path, file_mask: str) -> dt.datetime:

• store --since date in a separate file: if nothing to do, --since datetime becomes current datetime - file age offset (file age offsetdetailed in #1)
• if lftp has done something, --since becomes last file datetime - file age offset