waarp/waarp-all

This version is a major version for all Waarp Modules, previously being split.

Waarp Central

You will find in this web site some information on this Open Source project named Waarp, fully in Java.

For details, please go to Waarp Central Site https://waarp.github.io/Waarp/

The global license is GPL V3.

Waarp is a project that provides, among other packages, an open source massive file transfer monitor in Java. Its goal is to unify several protocols (FTP, SSH, HTTP and proprietary protocols) in an efficient and secured way. Its purpose is to enable bridging between several protocols and to enable dynamic pre or post action on transfer or other commands.

Packages

See unique package in Waarp-All

Support

Support is available through community and also through commercial support with the company named Waarp

• Installation and parameters
• Integration, additional development
• Support, maintenance, phone support

waarp-all's Issues

Build failed due to missing waarp theme for Sphinx

I'm trying to build (./mvnw -P jre11 -D skipTests package using Maven 3.8.1, OpenJDK 11, Debian 10) from branch v3.6.0 from fredericBregier fork (https://github.com/fredericBregier/Waarp-All/tree/v3.6.0) and I get an error related to Sphinx documentation generation:

[INFO] --- exec-maven-plugin:3.0.0:exec (SphinxHtml) @ WaarpPackaging ---
Sphinx v4.1.2 en cours d'exécution
chargement des traductions [fr]... fait
création du dossier de destinataire... fait

Theme error:
Le thème nommé 'waarp' n'a pas été trouvé (le fichier theme.conf est-il bien présent ?)
make: *** [Makefile:20: html] Error 2
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)

Based on Sphinx configuration file it seems that theme waarp is referenced but could not be found.

SHIFT not working with %* in waarp-r66client.bat

Problem is detected while executing send command on a windows machine.

OOME Error on pull request

It seems there is a memory leak on the R66 client.
While transmitting a big file (>400Mo) an OOME occurs.

client.log.2020-07-01.0.log

I suspect data packets are not discarded correctly.

Build failed if no Maven profile specified on command line

I'm trying to build from source using:

• AdoptOpenJDK 1.8.0_282
• Maven 3.6.3
• Debian Buster 10

mvn -P jre6 clean install lead to a successful build whereas mvn clean install failed (despite jre6 profile being enabled by default) with error:

[ERROR] Failed to execute goal org.codehaus.mojo:flatten-maven-plugin:1.2.5:flatten (flatten) on project WaarpCommon: 11 problems were encountered while building the effective model for Waarp:WaarpCommon:${revision}
[ERROR] [ERROR] 'dependencies.dependency.version' for com.h2database:h2:jar must be a valid version but is '${h2.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for org.mariadb.jdbc:mariadb-java-client:jar must be a valid version but is '${mariadb.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for mysql:mysql-connector-java:jar must be a valid version but is '${mysql.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for Waarp:WaarpDigest:jar must be a valid version but is '${waarp.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for commons-io:commons-io:jar must be a valid version but is '${commons.io.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for com.google.guava:guava:jar must be a valid version but is '${guava.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for org.dom4j:dom4j:jar must be a valid version but is '${dom4j.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for com.fasterxml.jackson.core:jackson-databind:jar must be a valid version but is '${jackson-databind.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for com.fasterxml.jackson.core:jackson-annotations:jar must be a valid version but is '${jackson.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for com.fasterxml.jackson.core:jackson-core:jar must be a valid version but is '${jackson.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar must be a valid version but is '${jackson.version}'. @ 

mvn help:active-profiles and mvn -P jre6 help:active-profiles gave exactly the same output.

If I run Maven with debug enable I can see some differences in flatten maven plugin debug logs:

• [DEBUG] Resolving dependencies of Waarp:Waarp-All:pom:3.5.2 if profile is specified as Maven command line argument
• [DEBUG] Resolving dependencies of Waarp:Waarp-All:pom:${revision} if profile is not specified as Maven command line argument

I did a similar test with a very minimal pom.xml that define artifact <version> using a property defined in a profile enabled by default and I experience similar behavior (even with latest flatten plugin version).

It seems that flatten plugin as several issue related to profile and property usage: https://github.com/mojohaus/flatten-maven-plugin/issues?q=is%3Aissue+is%3Aopen+profile

I guess the easy fix for this issue would be to update README.md to include -P jre6 (or maybe jre11 as version 11 is maybe now the most popular one) to avoid this issue.

Test execution hang in test1_Ftp4JSimple

In my attempt to build Waarp from sources available in https://github.com/fredericBregier/Waarp-All/tree/v3.6.0 I'm now facing an issue related to test execution.

Here is the latest logs I got when Waarp Ftp R66 client tests are executed:

Starting test: test1_Ftp4JSimple
10:29:26.614 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.testFtp4J(AbstractFtpClient.java:172) : First connexion
10:29:26.902 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.testFtp4J(AbstractFtpClient.java:182) : Create Dirs
.
10:29:26.923 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.testFtp4J(AbstractFtpClient.java:193) : Feature commands
SITE: true
SITE CRC: true
CRC: true
MD5: true
SHA1: true
DIGEST: true
10:29:26.944 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.testFtp4J(AbstractFtpClient.java:201) : Logout
10:29:26.947 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.testFtp4J(AbstractFtpClient.java:205) : Will start 1 Threads
10:29:27.117 [pool-1-thread-1] WARN  org.waarp.ftp.client.transaction.FtpClientThread - org.waarp.ftp.client.transaction.FtpClientThread.run(FtpClientThread.java:123) : T0 change mode passive

On another attempt I also got the following logs:

15:24:33.403 [pool-1-thread-1] WARN  org.waarp.ftp.client.transaction.FtpClientThread - org.waarp.ftp.client.transaction.FtpClientThread.run(FtpClientThread.java:123) : T0 change mode passive
...
15:24:33.863 [Data-3] WARN  org.waarp.ftp.core.data.handler.DataNetworkHandler - org.waarp.ftp.core.data.handler.DataNetworkHandler.channelActive(DataNetworkHandler.java:227) : DataChannel immediately closed since RETR is not ok at startup
15:24:34.062 [Data-4] WARN  org.waarp.ftp.core.data.handler.DataNetworkHandler - org.waarp.ftp.core.data.handler.DataNetworkHandler.channelActive(DataNetworkHandler.java:227) : DataChannel immediately closed since RETR is not ok at startup
15:24:34.067 [pool-1-thread-1] ERROR org.waarp.ftp.client.WaarpFtp4jClient - org.waarp.ftp.client.WaarpFtp4jClient.transferFile(WaarpFtp4jClient.java:505) : null: Retrieve operation not allowed
15:24:35.409 [Data-3] WARN  org.waarp.ftp.core.data.FtpTransferControl - org.waarp.ftp.core.data.FtpTransferControl.abortTransfer(FtpTransferControl.java:603) : Abort problem : No Command currently running
15:24:35.412 [Data-3] WARN  org.waarp.ftp.core.data.FtpTransferControl - org.waarp.ftp.core.data.FtpTransferControl.setTransferAbortedFromInternal(FtpTransferControl.java:700) : Command cancelled
15:24:35.592 [Data-4] WARN  org.waarp.ftp.core.data.FtpTransferControl - org.waarp.ftp.core.data.FtpTransferControl.abortTransfer(FtpTransferControl.java:603) : Abort problem : No Command currently running

I guess that the hang is probably related to awaitTermination that is configured to wait for 12000 seconds (200 minutes).

Also I think the calculation System.currentTimeMillis() - 120000 * 60; might be wrong because it should be * 1000 instead of * 60 to convert from seconds to milliseconds.

If I can provide more information in order to help debugging please let me know.

[WaarpR66] Possible reuse of closed database connection in HttpResponsiveSslHandler

Here are the logs:

04/05/2020/16:23:00.498 ERROR [org.waarp.common.database.DbPreparedStatement] [HttpWorker-27] org.waarp.common.database.DbPreparedStatement.createPrepareStatement(DbPreparedStatement.java:235) : [MYSERVER] SQL Exception createPreparedStatement from jdbc:postgresql://127.0.0.1:5432/waarp_r66:SELECT GLOBALSTEP,GLOBALLASTSTEP,STEP,RANK,STEPSTATUS,RETRIEVEMODE,FILENAME,ISMOVED,IDRULE,BLOCKSZ,ORIGINALNAME,FILEINFO,TRANSFERINFO,MODETRANS,STARTTRANS,STOPTRANS,INFOSTATUS,UPDATEDINFO,OWNERREQ,REQUESTER,REQUESTED,SPECIALID FROM RUNNER WHERE SPECIALID >= ? AND SPECIALID <= ? AND IDRULE LIKE '%%' AND ( REQUESTED LIKE '%%' OR REQUESTER LIKE '%%' ) AND OWNERREQ = 'MYSERVER' ORDER BY STARTTRANS DESC LIMIT 48 This connection has been closed.
04/05/2020/16:23:00.498 ERROR [org.waarp.common.database.DbSession] [HttpWorker-27] org.waarp.common.database.DbSession.error(DbSession.java:263) : [MYSERVER] SQLException: This connection has been closed. SQLState: 08003VendorError: 0
04/05/2020/16:23:00.498 WARN [org.waarp.openr66.protocol.http.adminssl.HttpResponsiveSslHandler] [HttpWorker-27] org.waarp.openr66.protocol.http.adminssl.HttpResponsiveSslHandler.setDbTaskRunnerJsonData(HttpResponsiveSslHandler.java:208) : [MYSERVER] OpenR66 Web Error SQL Exception createPreparedStatement: SELECT GLOBALSTEP,GLOBALLASTSTEP,STEP,RANK,STEPSTATUS,RETRIEVEMODE,FILENAME,ISMOVED,IDRULE,BLOCKSZ,ORIGINALNAME,FILEINFO,TRANSFERINFO,MODETRANS,STARTTRANS,STOPTRANS,INFOSTATUS,UPDATEDINFO,OWNERREQ,REQUESTER,REQUESTED,SPECIALID FROM RUNNER WHERE SPECIALID >= ? AND SPECIALID <= ? AND IDRULE LIKE '%%' AND ( REQUESTED LIKE '%%' OR REQUESTER LIKE '%%' ) AND OWNERREQ = 'MYSERVER' ORDER BY STARTTRANS DESC LIMIT 48

The same error happens with rules, hosts, export, etc., always in HttpResponsiveSslHandler

Waarp Password does not work

Hi,

On CentOS and Windows, waarp-password.sh / waarp-password.bat does not work correctly.

./bin/waarp-password.sh -ki ../waarp-certs/cryptokey.des -pwd adminpwd -po outputpwd.txt
Error: Could not find or load main class org.waarp.uip.WaarpPassword

After checking, the classe org.waarp.uip.WaarpPassword is not in share/lib/WaarpR66-3.3.0-jar-with-dependencies.jar
WaarpR66 Version : 3.3 / 3.2

Best Regards
TM

[WaarpGatewayFTP] The server exits as soon as it has started

The servers starts correctly, but shuts down right after.

I tried with several versions, the issue seems to have appeared somewhere between the switch from the old repo and the 3.1 version.

I attached a full debug log of the execution.
gwftp.log

Error during ExplainRequestIT test execution

I'm trying to build Waarp using Maven 3.9.1 (using Maven wrapper) and OpenJDK 11.0.18 on Debian Bullseye.

I'm running ./mvnw -P jre11 clean install and get the following error in the log (I can add the full build log if useful):

11:42:51.725 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.setUp3DbBeforeClass(ExplainRequestIT.java:279) : File /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/src/test/resources/it/scenario_big_file_limitbdw/R1/conf/server_1_SQLDB.xml exists? true
11:42:51.726 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.setUp3DbBeforeClass(ExplainRequestIT.java:283) : Dir /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66 exists? true
11:42:51.767 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.getServerConfigFile(ExplainRequestIT.java:204) : Build configFile
jdbc:postgresql://localhost:32918/test?loggerLevel=OFF
org.postgresql.Driver
postgresql
11:42:51.779 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.getServerConfigFile(ExplainRequestIT.java:256) : Config file created at /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml
11:42:51.780 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.getServerConfigFile(ExplainRequestIT.java:260) : Copy from /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml to /tmp/config.xml
11:42:51.781 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.getServerConfigFile(ExplainRequestIT.java:264) : Copy from /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml to /tmp/config.xml and return /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xmlExplainRequestIT
11:42:51.781 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.setUp3DbBeforeClass(ExplainRequestIT.java:290) : Config file found /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml
11:42:51.782 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.setUp3DbBeforeClass(ExplainRequestIT.java:308) : Could not find /tmp/config.xml or /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml
11:42:51.782 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.initiateDb(ExplainRequestIT.java:327) : File /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml exists? true
11:42:51.783 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.initiateDb(ExplainRequestIT.java:330) : File /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/src/test/resources/it/scenario_big_file_limitbdw/OpenR66-authent.xml exists? true
ERROR org.waarp.openr66.server.ServerInitDatabase  /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml -initdb -dir /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/src/test/resources/it/scenario_big_file_limitbdw/R1/conf -auth /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/src/test/resources/it/scenario_big_file_limitbdw/OpenR66-authent.xml return code: 1 pid: 99999999
11:42:52.304 [main] WARN  org.waarp.openr66.protocol.configuration.Configuration - org.waarp.openr66.protocol.configuration.Configuration.setTransferGuid(Configuration.java:2665) : Startup: Transfer Id will use SQL Sequence
11:42:52.334 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.initiateDb(ExplainRequestIT.java:327) : File /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/R2/conf/server_2.xml exists? true
11:42:52.335 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.initiateDb(ExplainRequestIT.java:330) : File /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/OpenR66-authent.xml exists? true
ERROR org.waarp.openr66.server.ServerInitDatabase  /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/R2/conf/server_2.xml -initdb -dir /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/R2/conf -auth /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/OpenR66-authent.xml return code: 1 pid: 99999999
11:42:52.631 [main] WARN  org.waarp.common.utility.WaarpLoggerListener - org.waarp.common.logging.AbstractWaarpLogger.log(AbstractWaarpLogger.java:168) : 
BUILD SUCCESSFUL
Total time: 0 seconds
Find server file: /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/R2/conf/server_2.xml
ERROR org.waarp.openr66.server.R66Server  /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/R2/conf/server_2.xml return code: 0 pid: 390356
11:42:52.912 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.startServer(ExplainRequestIT.java:360) : Start Done: 390356
11:42:53.438 [main] ERROR org.waarp.common.database.DbRequest - org.waarp.common.database.DbRequest.select(DbRequest.java:127) : [server1] SQL Exception Request:SELECT * FROM  CONFIGURATION  ERROR: relation "configuration" does not exist
  Position : 16
11:42:53.450 [main] ERROR org.waarp.common.database.DbConstant - org.waarp.common.database.DbConstant.error(DbConstant.java:69) : [server1] SQLException: ERROR: relation "configuration" does not exist
  Position : 16 SQLState: 42P01 VendorError: 0
11:42:53.452 [main] WARN  org.waarp.openr66.configuration.FileBasedConfiguration - org.waarp.openr66.configuration.FileBasedConfiguration.loadDatabase(FileBasedConfiguration.java:1670) : [server1] Database is not yet initiated: run ServerInitDatabase -initdb first : SQL Exception Request:SELECT * FROM  CONFIGURATION 
11:42:54.272 [main] WARN  org.waarp.openr66.configuration.FileBasedConfiguration - org.waarp.openr66.configuration.FileBasedConfiguration.loadServerParam(FileBasedConfiguration.java:491) : [server1] Multiple Monitor configuration unactive
11:42:54.356 [main] ERROR org.waarp.openr66.configuration.FileBasedConfiguration - org.waarp.openr66.configuration.FileBasedConfiguration.setConfigurationServerFromXml(FileBasedConfiguration.java:2324) : [server1] Cannot find Authentication for current host
11:42:54.356 [main] ERROR org.waarp.openr66.server.R66Server - org.waarp.openr66.server.R66Server.initialize(R66Server.java:84) : [server1] Needs a correct configuration file as first argument
11:42:54.357 [main] ERROR org.waarp.openr66.server.R66Server - org.waarp.openr66.server.R66Server.main(R66Server.java:66) : [server1] Cannot start Server OpenR66 for server1
ERROR Cannot start Server OpenR66 for server1
####################### NO R66 SERVER RUNNING ###################
ERROR Exit end of Shutdown
[INFO] Running org.waarp.openr66.protocol.it.ExplainRequestIT
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.023 s <<< FAILURE! - in org.waarp.openr66.protocol.it.ExplainRequestIT
[ERROR] org.waarp.openr66.protocol.it.ExplainRequestIT  Time elapsed: 0.013 s  <<< ERROR!
java.lang.NullPointerException
	at [email protected]/org.waarp.openr66.protocol.it.ExplainRequestIT.setUpBeforeClass(ExplainRequestIT.java:189)
	at [email protected]/org.waarp.openr66.protocol.it.ExplainRequestIT.setup(ExplainRequestIT.java:152)

What is weird is that line 189 in ExplainRequestIT.java is setUpBeforeClassClient(); so I don't understand how it can trigger a NullPointerException.

[R66] self-transfers are not saved in the database unless it fails

When I try to do a transfer from an agent to itself, the transfer is not saved in the database.

For example, of these two transfers (from server1 to server1):

Only the second one exists in the database:

I only observed this for "transfers to self".

Is there any commercial support available for this project

Is there any commercial support available for this project

[R66] LogExtendedExport does not work anymore

The command was functional in 3.0.x versions, but on 3.4 dev branch, it does not seem to send the request to the server.

Here are the logs when i try this command:

14:51:28.516 [main] WARN org.waarp.openr66.protocol.configuration.Configuration - org.waarp.openr66.protocol.configuration.Configuration.pipelineInit(Configuration.java:766) : [server2] Server Thread: 8 Client Thread: 80 Runner Thread: 1000
14:51:28.517 [main] INFO org.waarp.openr66.protocol.configuration.Configuration - org.waarp.openr66.protocol.configuration.Configuration.pipelineInit(Configuration.java:772) : [server2] Current launched threads: 8
14:51:28.558 [main] DEBUG org.waarp.openr66.server.LogExtendedExport - org.waarp.openr66.server.LogExtendedExport.run(LogExtendedExport.java:187) : [server2] ExtendedLogCommand: {"@class":"org.waarp.openr66.protocol.localhandler.packet.json.LogJsonPacket","comment":null,"requestUserPacket":0,"purge":false,"clean":false,"statuspending":false,"statustransfer":false,"statusdone":true,"statuserror":false,"rule":null,"request":null,"start":null,"stop":1594850400000,"startid":null,"stopid":null}
14:51:28.558 [main] INFO org.waarp.openr66.server.LogExtendedExport - org.waarp.openr66.server.LogExtendedExport.run(LogExtendedExport.java:189) : [server2] *** before tryconnect
14:51:28.559 [main] DEBUG org.waarp.openr66.protocol.networkhandler.NetworkTransaction - org.waarp.openr66.protocol.networkhandler.NetworkTransaction.closeAll(NetworkTransaction.java:1371) : [server2] close All Network Channels
14:51:28.761 [main] DEBUG org.waarp.common.database.DbAdmin - org.waarp.common.database.DbAdmin.closeAllConnection(DbAdmin.java:396) : [server2] Close (all) Db Conn: aeaaaaaap3ilnhaiplsac42xvyaqkaaaae

From what I can see, and with a few more testing, the code for the command
(LogExtendedExport.run()) is executed until line
189:

LocalChannelReference localChannelReference =
        AbstractTransfer.tryConnect(host, newFuture, networkTransaction);

I don't know why, tryConnect does not seem to be executed (its first line is a logger call, which does not appear in the logs).

The REST v2 API ignores a user's admin attribute in authorization

In the REST v1 API, a user with admin = true as a de facto FULLADMIN role.

However, in the REST v2 API, a user with admin = true as a de facto NOACCESS role.

The output of client does not escape values according to the target format

Some output values can contain XML-like data.
When we ask the command to format the output in XML with -xml, this produce a valid XML structure, but it does not match the intent.

For example, with the output for a failed transfer, I have the following XML (the formatting has been added by me to highlight what I mean):

<xml>
  <command>DirectTransfer</command>
  <args>etc/conf.d/server2/client.xml -file world.txt -to server1 -rule error -xml </args>
  <status>2</status>
  <statusTxt>Transfer in status: FAILED with no Transfer Id</statusTxt>
  <remote>server1</remote>
  <error>
    ErrorPacket:(3:E) PreTask in error: Runner is failed: 
    <STATUS>-1</STATUS>
    <ERROR>will fail </ERROR>
  </error>
</xml>

Though it is valid, <STATUS> and <ERROR> are part of <error> and not tags of the main XML document.
The output should use entity refs to protect the content of the <error> message:

<xml>
  <command>DirectTransfer</command>
  <args>etc/conf.d/server2/client.xml -file world.txt -to server1 -rule error -xml </args>
  <status>2</status>
  <statusTxt>Transfer in status: FAILED with no Transfer Id</statusTxt>
  <remote>server1</remote>
  <error>
    ErrorPacket:(3:E) PreTask in error: Runner is failed: 
    &lt;STATUS&gt;-1&lt;/STATUS&gt;
    &lt;ERROR&gt;will fail &lt;/ERROR&gt;
  </error>
</xml>

edit:
I realised by reviewing #67 that the same problem exists for other output formats.

Here is an example with -csv. The culprits are the semicolon (;) between "will" and "fail" and the new line which are not protected, hence adding an extra column in the first line, and a second record in the second line:

command;args;status;statusTxt;remote;error
DirectTransfer;etc/conf.d/server2/client.xml -file world.txt -to server1 -rule error -csv ;2;Transfer in status: FAILED with no Transfer Id;server1;ErrorPacket:(3:E) PreTask in error: Runner is failed: <STATUS>-1</STATUS><ERROR>will ;fail
always

The output should be:

command;args;status;statusTxt;remote;error
DirectTransfer;etc/conf.d/server2/client.xml -file world.txt -to server1 -rule error -csv ;2;Transfer in status: FAILED with no Transfer Id;server1;"ErrorPacket:(3:E) PreTask in error: Runner is failed: <STATUS>-1</STATUS><ERROR>will ;fail
always"

Escaping issue in rules whose path contain "\"

Steps to reproduce

1. Go to the rule page in the admin interface and select a rule to edit.
2. Change a path to contain \ (for example C:\Data\)
   
3. Update the rule.
4. In the resulting list, the path contains \\ instead of \
   
5. When re-editing the rule, the \\ are kept
   
6. Saving the rule without fixing the paths will re-double the \, and so on
   

testSysmonGetLoadAverage failed

In our CI execution of testSysmonGetLoadAverage failed with error message: java.lang.AssertionError: Max > current: 1.0 >? 1.0. The CI is building the v3.6.1 branch from https://github.com/fredericBregier/
You can see the details here: https://gitlab.ow2.org/ow2/oscar/waarp/-/jobs/65315#L424
CI passed successfully on Aug 24, 2021 and failed on Aug 29, 2021 so based on the list of commits I'm assuming that the issue might have been introduced by the commit 086c6b7.

[R66] some actions are not shown in the system tab of the https admin interface

In the https admin interface, in the system tab, the last row of buttons only shows some actions ("shutdown", "restart", "block" and "export config") when another action has been activated on the page (for example "language"):

Démarrage avec 2 "r" dans la doc

Petite faute d'orthographe dans la documentation, voir : https://doc.waarp.org/waarp-r66/latest/fr/
Le titre du 1er chapitre devrait être "Démarrage" et non pas "Démarage" ...

[R66] XMLRuleDAO select, find and exists operations only work on internal cache

To avoid lookups on disk every time a Rule is selected, XMLRuleDAO has an internal cache, which is good.

However, select, find and exists operations only works on the internal cache (as opposed to XMLHostDAO which falls bach on disk search if nothing id found in the internal cache).

As the internal cache if only filled when getAll() is called, there are some situations where select, find and exists never find rules that do exists.
As a side effect, the file watcher is not working without database: SpooledDirectoryTransfer (and its execution tree) never runs XMLRuleDAO.getAll() durining its initialization, so the individual lookups for each spooldirectory's rule always fail.

Transfer listing does not load if a rule has been deleted

In the Admin HTTPS interface, if a transfer uses a rule that has been removed, the transfer listing does not show the transfer, but an error message.

Steps to reproduce

• create a rule "foo"
• do a transfer with the rule "foo"
• delete the rule "foo"
• go to the admin interface and navigate to the transfer listing page

Expected outcome

the transfer that used the rule "foo" shows in the list of transfers

Actual outcome

The https interface shows this error:

!ErrorCode.17!: Rule cannot be found for DbTaskRunner: {"@model":"DbTaskRunner","SPECIALID":-9223372036854775795,"RETRIEVEMODE":false,"IDRULE":"test","MODETRANS":1,"FILENAME":"/in/-9223372036854775795_3185345321839916308_foo.txt","ORIGINALNAME":"/out/foo.txt","FILEINFO":"noinfo","ISMOVED":false,"BLOCKSZ":65536,"OWNERREQ":"server","REQUESTER":"client","REQUESTED":"server","TRANSFERINFO":"{\"ORIGINALSIZE\":12}","GLOBALSTEP":4,"GLOBALLASTSTEP":4,"STEP":0,"STEPSTATUS":"O ","INFOSTATUS":"O ","RANK":1,"STARTTRANS":1587989702508,"STOPTRANS":1587989702609,"UPDATEDINFO":6,"ORIGINALSIZE":-1}

The logs contains only this warning :

28/04/2020/09:58:57.783 WARN [org.waarp.openr66.protocol.http.adminssl.HttpResponsiveSslHandler] [HttpWorker-12] org.waarp.openr66.protocol.http.adminssl.HttpResponsiveSslHandler.setDbTaskRunnerJsonData(HttpResponsiveSslHandler.java:208) : [server] OpenR66 Web Error Rule cannot be found for DbTaskRunner: {"@model":"DbTaskRunner","SPECIALID":-9223372036854775795,"RETRIEVEMODE":false,"IDRULE":"test","MODETRANS":1,"FILENAME":"/in/-9223372036854775795_3185345321839916308_foo.txt","ORIGINALNAME":"/out/foo.txt","FILEINFO":"noinfo","ISMOVED":false,"BLOCKSZ":65536,"OWNERREQ":"server","REQUESTER":"client","REQUESTED":"server","TRANSFERINFO":"{\"ORIGINALSIZE\":12}","GLOBALSTEP":4,"GLOBALLASTSTEP":4,"STEP":0,"STEPSTATUS":"O  ","INFOSTATUS":"O  ","RANK":1,"STARTTRANS":1587989702508,"STOPTRANS":1587989702609,"UPDATEDINFO":6,"ORIGINALSIZE":-1} 

Proposition for unique transfer ID

Given two partners exchanging transfers if one of the transfer IDs is reset (DB purge, ...) new transfers cannot be added as the partners who did not reset its IDs will refuse with the reason the TransferID already exists for the given owner/requester/requested.

A solution would be to use unix epoch in nanoseconds for transfer IDs (instead of DB increments) , this would ensure uniqueness of IDs given a unique partners does not overload with transfers. The IDs would have a expiration date in year 2554 which should be a comfortable margin.

Will Waarp support the PeSIT protocol?

Hello,

I saw on Waarp-gateway's GitLab repository a branch called "Add sync & seek methods to the pipeline" created by a user who plans to add compatibility for the PeSIT protocol.

Since this branch hasn't been updated for six months, I wanted to know if the PeSIT protocol is still planned for Waarp or not. If it's the case, is there a roadmap or a release date?

Allow absolute path in <directory> config

Currently paths in the directory group of the config files are used as follow:
serverhome + value (value among in, out, work, arch, conf)
This prevent the use of absolute path in the value part of the configuration.
A solution would be to modify the getters of the configuration java like this:

public String getConfigPath() {
  File conf = new File(configPath);
  if (conf.isAbsolute()) {
    return configPath;
  }
  return getBaseDirectory() + configPath;
}

@fredericBregier what do you think ?

[R66] the formatting of the output of the send commands is broken

It seems there is no support for the formatting options of the send comands: -xml, -json, -property, etc.

➜ ./bin/waarp-r66client.sh client1 send -property -file ./file.dat -to server1 -rule push
usage: Transfer
 -block <arg>   Specify the block size
 -delay <arg>   Specify the delay time as an epoch time or '+' a delay in
                ms
 -file <arg>    Specify the file path to operate on
 -id <arg>      Specify the id of transfer
 -info <arg>    Specify the transfer information
 -logWarn       Specify to log final result as Warn if OK
 -md5           Specify the option to have a hash computed for the
                transfer
 -nofollow      Specify if the transfer should not integrate a FOLLOW id
 -nolog         Specify to not log anything included database once the
                transfer is done
 -notlogWarn    Specify to log final result as Info if OK
 -rule <arg>    Specify the Rule
 -start <arg>   Specify the start time as yyyyMMddHHmmss
 -to <arg>      Specify the requested Host
Wrong initialization

[R66] Listing pages are broken in the admin interface

FollowId addition breaks the json included in the html source:

I can only guess it comes from DbTaskRunner.getJson() which uses arrayNode.toString() to generate JSON.

but according to jackson doc:

Method that will produce developer-readable representation of the node; which
may or may not be as valid JSON. If you want valid JSON output (or output
formatted using one of other Jackson supported data formats) make sure to use
ObjectMapper or ObjectWriter to serialize an instance, for example:

String json = objectMapper.writeValueAsString(rootNode);

I imagine it also would remove the need to do "manual" cleaning of the json generated.

Usability issues in R66 admin interface

In the "Transfers > Listing" and "Transfers > Cancel/restart" pages:

• sorting does not work for the specialid column
• The "clear" button does not work if the list has already been filtered (a click on "clear" works just after having typed text though)

FtpClient2TLSNativeTest test failed

I'm trying to build Wraap from source for and with Java 11. I'm doing that using GitLab CI (see my gitlab-ci.yml). It basically get the branch v3.6 and try to build it by running mvn --batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true -P jre11 clean org.jacoco:jacoco-maven-plugin:prepare-agent test org.jacoco:jacoco-maven-plugin:report (with Maven 3.8.1 and OpenJDK 11).

Test failed but the logs does not give much information about the cause of the failure:

08:32:37.811 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.stopServer(AbstractFtpClient.java:96) : Will shutdown from client
08:32:38.084 [Command-5] WARN  org.waarp.ftp.core.command.internal.INTERNALSHUTDOWN - org.waarp.ftp.core.command.internal.INTERNALSHUTDOWN.exec(INTERNALSHUTDOWN.java:82) : Shutdown...
SHUTDOWN: System shutdown
08:32:38.089 [Command-5] WARN  org.waarp.ftp.core.utils.FtpChannelUtils - org.waarp.ftp.core.utils.FtpChannelUtils.exit(FtpChannelUtils.java:343) : Exit: Give a delay of 500 ms
08:32:38.090 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.stopServer(AbstractFtpClient.java:120) : Will stop server
ERROR Exit System
477833 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.953 s <<< FAILURE! - in org.waarp.ftp.client.FtpClient2TLSNativeTest
477834 [ERROR] org.waarp.ftp.client.FtpClient2TLSNativeTest.test0_0FtpApacheZ  Time elapsed: 0.373 s  <<< FAILURE!
java.lang.AssertionError: No KO expected:<0> but was:<1>
	at org.junit.Assert.fail(Assert.java:89)

You can view the CI job log here: https://gitlab.ow2.org/ow2/oscar/waarp/-/jobs/63217

Maybe this is related to #99 ?

[R66] Commander might be blocked in some cases

The commander lists the tansfers to start, and loops on them.
In the loop, there are two branches with continue statements, but the loop index is not incremented before the continue. This results in infinite loops when a transfer enters one of these branches.

Waarp-All/WaarpR66/src/main/java/org/waarp/openr66/commander/Commander.java

Lines 437 to 467 in 07caa8a

	while (i < tasks.length) {
	if (WaarpShutdownHook.isShutdownStarting()) {
	logger.info("Will not start transfers, server is in shutdown.");
	return;
	}
	final DbTaskRunner taskRunner = tasks[i];
	logger.debug("get a task: {}", taskRunner);
	// Launch if possible this task
	final String key =
	taskRunner.getRequested() + ' ' + taskRunner.getRequester() +
	' ' + taskRunner.getSpecialId();
	if (Configuration.configuration.getLocalTransaction()
	.getFromRequest(key) != null) {
	// already running
	continue;
	}
	if (taskRunner.isSelfRequested()) {
	// cannot schedule a request where the host is the requested host
	taskRunner.changeUpdatedInfo(UpdatedInfo.INTERRUPTED);
	try {
	taskRunner.update();
	} catch (final WaarpDatabaseNoDataException e) {
	logger.warn("Update failed, no transfer found");
	}
	continue;
	}
	internalRunner.submitTaskRunner(taskRunner);
	i++;
	totalRuns++;
	}

Moving the i++ from line 465 to line 443 should fix this)

The JRE never exits when a DirectTransfer receives a "HostId not allowed" error

The scenario to reproduce the error is the following :

an R66 client sends a file to an R66 server, which has no entry for the client in its partner database, with a DirectTransfer (I suppose it is the same for other types of transfers).
The client receives an error packet A, closes its network and database connections, but never exits.

I attached the debug logs for such a transfer.

Can it be included in the 3.3.2 version ?

client.log

[R66] ClassCastException thrown by EXECOUTPUT tasks the the command failed

Here is the full stacktrace:

30/07/2020/12:47:31.051 DEBUG [org.waarp.openr66.protocol.localhandler.LocalServerHandler] [Handler-1] org.waarp.openr66.protocol.localhandler.LocalServerHandler.exceptionCaught(LocalServerHandler.java:370) : [server1] Exception and isFinished: false
java.lang.ClassCastException: java.lang.Thread cannot be cast to org.waarp.openr66.context.task.AllLineReader
	at org.waarp.openr66.context.task.ExecOutputTask.finalizeFromError(ExecOutputTask.java:204)
	at org.waarp.openr66.context.task.AbstractExecTask$ExecuteCommand.invoke(AbstractExecTask.java:552)
	at org.waarp.openr66.context.task.ExecOutputTask.run(ExecOutputTask.java:127)
	at org.waarp.openr66.database.data.DbTaskRunner.runNextTask(DbTaskRunner.java:2718)
	at org.waarp.openr66.database.data.DbTaskRunner.runNext(DbTaskRunner.java:2796)
	at org.waarp.openr66.database.data.DbTaskRunner.run(DbTaskRunner.java:2844)
	at org.waarp.openr66.database.data.DbTaskRunner.finalizeTransfer(DbTaskRunner.java:3035)
	at org.waarp.openr66.context.R66Session.setFinalizeTransfer(R66Session.java:953)
	at org.waarp.openr66.protocol.localhandler.TransferActions.endTransferR(TransferActions.java:1168)
	at org.waarp.openr66.protocol.localhandler.TransferActions.fromEndTransferSToTransferR(TransferActions.java:1048)
	at org.waarp.openr66.protocol.localhandler.TransferActions.endTransfer(TransferActions.java:1033)
	at org.waarp.openr66.protocol.localhandler.LocalServerHandler.channelRead1(LocalServerHandler.java:265)
	at org.waarp.openr66.protocol.localhandler.LocalServerHandler.channelRead0(LocalServerHandler.java:107)
	at org.waarp.openr66.protocol.networkhandler.NetworkServerHandler.channelRead0(NetworkServerHandler.java:392)
	at org.waarp.openr66.protocol.networkhandler.NetworkServerHandler.channelRead0(NetworkServerHandler.java:63)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.lang.Thread.run(Thread.java:748)

Steps to reproduce :

create a rule that starts the script fail.sh, which always ends in error :

Content of the rule:

<rule>
  <idrule>error</idrule>
  <comment>The default rule send rule (extended version)</comment>
  <mode>1</mode>
  <rpretasks>
    <tasks>
    </tasks>
  </rpretasks>
  <rposttasks>
    <tasks>
      <task>
        <type>EXECOUTPUT</type>
        <path>./bin/fail.sh</path>
        <delay>60000</delay>
      </task>
    </tasks>
  </rposttasks>
  <rerrortasks>
    <tasks>
    </tasks>
  </rerrortasks>
  <spretasks>
    <tasks>
    </tasks>
  </spretasks>
  <sposttasks>
    <tasks>
    </tasks>
  </sposttasks>
  <serrortasks>
    <tasks>
    </tasks>
  </serrortasks>
</rule>

content of the script:

#!/usr/bin/env bash

echo will fail
exit 34