[BUG] SFTP Include All Subdirectories redundant requests

[michaelmarcuccio]

[BUG] SFTP Include All Subdirectories redundant requests

[jonbartels]

I think the relevant code is -

connect/server/src/com/mirth/connect/connectors/file/FileReceiver.java

Line 218 in e66d112

if (connectorProperties.isDirectoryRecursion()) {

and

connect/server/src/com/mirth/connect/connectors/file/filesystems/SftpConnection.java

Line 204 in e66d112

public List<String> listDirectories(String fromDir) throws Exception {

I'm not sure whats going on but see how the Sftpconnection does a cwd then an ls? I wonder if you're seeing repeated cd calls then the ls?

Mirth Connect uses JSch for SFTP. Are there any similar issue reports for JSch? In other words, is this an MC issue or a JSch issue?

Turning up logging for JSch might also show what commands are being issued.

[michaelmarcuccio]

With Mirth upgrading to log4j2, how do you enable JSch logging?
Service won't start using the old way: logger.JSch = DEBUG

[jonbartels]

https://forums.mirthproject.io/forum/mirth-connect/support/12545-sftp-file-reader-connection-log JSch logging is weird. You have to add a deploy script.

[michaelmarcuccio]

I got the logs working, but after researching a bit it turns out JSch logs do not log anything past auth info.
"It can get you a lot of info about the connection process (key exchange and such). But there is practically no such thing as debug output for the core functionality after authentication, at least for SFTP." This is also what I am seeing in the debug level logs. No info on actual transactions.

Maybe I am just mistaken on the functionality of SFTP polling. Is it logical for a single folder to take 1 second to poll but a folder and 3 subdirectories 4 seconds total to poll? I was under the assumption that it would be less than O(n).

[jonbartels]

Is your log output really getting a single polling event or is it getting multiple polling events?

[michaelmarcuccio]

If you are asking about the JSch logs from the Mirth server, there are many many rows of logging the sftp connection/security info. But there is no logging of the actual request such as ls/cd.

[michaelmarcuccio]

This may just be a weird way my SFTP server logs requests. If anyone else has logs from their SFTP server of Mirth polling subdirectories that would help.

[jonbartels]

@michaelmarcuccio - I haven't had time to try it yet but could this be verified with a docker-compose set of containers with MC and an SFTP server?

Launch both containers and have MC chatter with the SFTP server then look at the logs from each.

[michaelmarcuccio]

@jonbartels yeah that would work, I guess it all depends on how good the sftp server's logs are to maybe get more details.