Comments (8)
Thankyou @puellanivis for your reply and explanation throughout this thread.
I did a more debugging by writing a go test with goroutines and then found I was passing *sftp.Client by value instead of by reference. This was causing breaking of the flow in case of multithreading.
Thankyou for your time. Will make a note of sharing the method signature as well next time to catch such mistakes earliest.
from sftp.
I’m not clear on why you are generating multiple clients.
The sftp.NewClient()
should work the same as the ssh.Dial()
for the duration of the life of the program. (Unless there is a disconnect event, but you should handle that case with a complete reconnect with a new ssh.Dial()
)
from sftp.
Thankyou for reply @puellanivis.
In my initial code, I created ssh.Dial() and sftp.NewClient() just once both in Start method. and used this client in a code (for e.g client.OpenFile() ) which gets called from engine multiple times based on number of jobs spawned. if the job spawned is just one, client.OpenFile() works fine but when two parallel jobs are spawned I get connection lost error.
It looks like there was some error which caused the client to close the connection (connection was not closed by my Stop method). Why exactly I am not able to use same client for two parallel requests? or do I need to handle in some different way?
Let me also try to explain this as usecase-
I have start method having ssh.Dial and NewClient()
I have a stop method where I close client and ssh connection
Case 1: I have one flow -> having SFTP download operation followed by SFTP upload operation
This works fine.
Case 2 : I have two flows
Flow1 has -> SFTP download followed by SFTP upload
Flow2 is exact copy of Flow1
Start stop is called just once but these two flows where client.OpenFile is used is called in parallel not after one another. And in that case I get Connection Lost error. Some error must have caused this connection to close (and not my stop method).
How can I handle this?
from sftp.
Your Case 2 scenario should work just fine. I would however, have to see some source code of a reproduction to know what you might be doing wrong.
As an example, #572 is opening 1000 files all at the same time for writing, and while it does hit a bug, that’s only because it calls the file close operation twice, and we’re addressing it.
from sftp.
Hello @puellanivis , I wasn't opening the client with ClientOption like UseConcurrentReads, UseConcurrentWrites. But even after I did now, I am still getting same error.
Let me share the snippet -
//start method
config := ssh.ClientConfig{
User: s.User,
Auth: []ssh.AuthMethod{
ssh.PublicKeys(signer),
},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
}
addr := fmt.Sprintf("%s:%d", s.Host, s.Port)
conn, _ := ssh.Dial("tcp", addr, &config)
//defer conn.Close() is commented on purpose and close is moved to Stop method called when engine is shut
client, _ := sftp.NewClient(conn)
//defer client.Close() is commented on purpose and close is moved to Stop method called when engine is shut
//stop method
client.Close()
conn.Close()
//upload operation -> file to file transfer ---this method gets called in parallel by engine if I spawn more than one job
srcFile, _ := os.Open(localFileName)
defer srcFile.Close()
//file on sftp server
var dstFile *sftp.File
if overwrite {
dstFile, err = client.OpenFile(remoteFileName, (os.O_TRUNC | os.O_CREATE | os.O_WRONLY))
} else {
dstFile, err = client.OpenFile(remoteFileName, (os.O_APPEND | os.O_CREATE | os.O_WRONLY))
}
if err != nil {
fmt.Println(err.Error())
}
defer dstFile.Close()
bytes, err := io.Copy(dstFile, srcFile)
if err != nil {
fmt.Println(err.Error())
}
fmt.Println(bytes)
Error I am getting is upload operation - connection lost at client.OpenFile
Error when I shut the engine and stop method is called -
Error closing SFTP client : EOF
What am I doing incorrectly?
from sftp.
The UseConcurrentReads
and UseConcurrentWrites
were irrelevant to the bug in the linked to issue, as they were only ever transferring 14-bytes at a time, well below the maximum size of an SFTP data packet.
I still can’t really say what is going wrong, I need to see more code, specifically, the parallelizing code, particularly since you’re only seeing issues if you call it in parallel.
What server are you trying to talk to?
from sftp.
Parallel calling code is more of a engine code and I get a callback from it. I do no have access to that.
I am connecting with jscape sftp server, .rebex tiny sftp server.
from sftp.
I am unsure how I could proceed towards helping you without a grasp for how the parallel calling code works… there is nothing wrong with the code you have posted, and there is no reason for why it should be acting the way you are describing, except or unless it’s closing the client?
Although maybe, 🤔 does the same behavior persist if you try it against an openssh sftp server? Perhaps there is a subtle incompatibility of our packet encodings/decodings?
from sftp.
Related Issues (20)
- Question about getwd() method HOT 1
- IETF Link has rotted HOT 2
- Support for the Afero interface HOT 5
- mandatory workDir even for absolute paths HOT 1
- Unable to delete the selected user: Not Found HOT 3
- Change current working directory? HOT 2
- proposal/exposeRemoveFile HOT 2
- Request attributes are not being propagated for SSH_FXP_OPEN operation HOT 5
- Should Seek() io.SeekStart and io.SeekCurrent do the different things? HOT 1
- sftp client hang when exit in sftp server example HOT 2
- Connection lost when using library with limited bandwidth HOT 5
- In Windows, `Server` should allow a pseudo-directory for `/` which enumerates available drives HOT 22
- How can I perform sftp download and upload feature for multiple files at root directory and within nested subfolder using https://github.com/pkg/sftp HOT 3
- close file double time sometimes cause copy problem wh HOT 10
- how to handle binary and ascii file transfer to and from sftp server? HOT 1
- The performance of transferring file is bad HOT 1
- Does this package Support z/OS (s390x) or not? HOT 2
- Performance is much slower than a Java implementation when running in z/OS HOT 5
- Proposal: Standardized FileInfo Structure. RE: z/OS performance investigation HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sftp.