studio-b12/gowebdav

A golang WebDAV client library and command line tool.

gowebdav's Issues

Can't authorize at server

Hello.
I found a bug about authorization.

At requests.go:req(...) authorization processed after copying headers from WebDAV client to HTTP client, not before. So authorization is impossible.

I created pull request which solves this issue -- #13

This is a library usage question，about ContentType.

I saw that the Type File Struct has a field of ContentType that can be used, but this is just a struct and cannot get the ContentType of the file. Maybe I'm not very familiar with golang and would like to ask how to get it?

type File struct {
	path        string
	name        string
	contentType string
	size        int64
	modified    time.Time
	etag        string
	isdir       bool
}

func (f File) ContentType() string {
	return f.contentType
}

[WIP] Supports upload files via WriteCloser

Supports write file via io.WriteCloser i.e., create a function that returns io.WriteCloser.

Could be as follows:
#63 func (c *Client)CreateWdFile(path string,r io.Reader) (writer io.WriteCloser, err error) {}

401 Unauthorized when trying to login to Apache HTTPS Server

Hello, studio-b12!

I use following software:
OS: Windows 10 x64
Golang: 1.10.1

I trying to use your app to connect to my WebDav sever (Apache HTTPS Server), via following commands in cmd:

set ROOT=mywebdav.lan
set USER=user
set PASSWORD=password

gowebdav -X LS /user

and expect to see my files and folders in /user folder:

folder-1
folder-2
folder-3
file.txt

but instead I got following:

ReadDir /data/: 401 Unauthorized - PROPFIND /data/

My server works good and I have no problem to connect to it via Google Chrome.

How can I fix this?
Thanks in advance.

Improve README

I propose to:

1. create separate README file for gowebdav library and its cmd client
2. remove godoc text from README file

improve authentication

fix #14 in commit b45378c changed the behaviour of the req() method. the auth checks were moved from Connect() to req().

how to improve the code to keep the modification inside the client, because it's a client improvement and to keep the request as small as possible.

suggestions and discussion are welcome!

File upload uploads files with 0 bytes - No error given

Describe the bug
I discovered a bug in the file upload that has some relation to #20.
When uploading a file to Nextcloud, the command succeeds and exits without any error, but the uploaded file has a size of 0 bytes.

Software

• OS: Windows (also tested on Linux)
• Golang: go1.14.6 windows/amd64
• Version: current master

To Reproduce

1. checkout this project
2. Setup your environment with the ROOT, USER and PASSWORD
3. Pick a file you want to upload
4. Upload the file using the command

gowebdav -X PUT /webdav_test.png .\Chart_KAD.PNG

Put: .\Chart_KAD.PNG -> /webdav_test.png`

7. Look at the Nextcloud instance to see that the actual uploaded file-size is 0 bytes.

Apache access log (nextcloud)

80.187.108.153 - - [12/Aug/2020:12:32:26 +0200] "PUT /remote.php/dav/files/RAYs3T/webdav_test.png HTTP/1.1" 401 557
80.187.108.153 - - [12/Aug/2020:12:32:27 +0200] "PUT /remote.php/dav/files/RAYs3T/webdav_test.png HTTP/1.1" 201 -

Expected
The file should upload completly or fail with an error

Additional context
I initially discovered this issue via PhotoPrism (which uses this client for the uploads).
At least one other user is facing the same issue (photoprism/photoprism#443).

panic due to concurrent map writes

I'm sometimes getting this panic when running multiple requests in parallel using a single webdav client:

	/usr/local/Cellar/go/1.14.6/libexec/src/runtime/panic.go:1116 +0x72 fp=0xc0001d9448 sp=0xc0001d9418 pc=0x10351d2
runtime.mapassign_faststr(0x1c15400, 0xc00001f0b0, 0x1ce5cd4, 0xd, 0x64)
	/usr/local/Cellar/go/1.14.6/libexec/src/runtime/map_faststr.go:291 +0x3de fp=0xc0001d94b0 sp=0xc0001d9448 pc=0x101529e
net/textproto.MIMEHeader.Set(...)
	/usr/local/Cellar/go/1.14.6/libexec/src/net/textproto/header.go:22
net/http.Header.Set(...)
	/usr/local/Cellar/go/1.14.6/libexec/src/net/http/header.go:37
github.com/studio-b12/gowebdav.(*BasicAuth).Authorize(0xc0005585c0, 0xc00001f0e0, 0x1ce1500, 0x8, 0x1cdc5ce, 0x1)
	/Users/jarek/go/pkg/mod/github.com/studio-b12/[email protected]/basicAuth.go:32 +0x1d0 fp=0xc0001d9578 sp=0xc0001d94b0 pc=0x18d1c80
github.com/studio-b12/gowebdav.(*Client).req(0xc00001f0e0, 0x1ce1500, 0x8, 0x1cdc5ce, 0x1, 0x1e96e40, 0xc0002807a0, 0xc0001d97a0, 0x0, 0x0, ...)
	/Users/jarek/go/pkg/mod/github.com/studio-b12/[email protected]/requests.go:28 +0x283 fp=0xc0001d9740 sp=0xc0001d9578 pc=0x18d6353
github.com/studio-b12/gowebdav.(*Client).propfind(0xc00001f0e0, 0x1cdc5ce, 0x1, 0x26cdc00, 0x1d1eebf, 0xcb, 0x1acf280, 0xc00028d2f0, 0xc0001d98b0, 0x0, ...)
	/Users/jarek/go/pkg/mod/github.com/studio-b12/[email protected]/requests.go:93 +0x114 fp=0xc0001d9808 sp=0xc0001d9740 pc=0x18d7544
github.com/studio-b12/gowebdav.(*Client).ReadDir(0xc00001f0e0, 0x1cdc5ce, 0x1, 0x1cdc5ce, 0x1, 0x1cdc5ce, 0x1, 0xc0002a8520)
	/Users/jarek/go/pkg/mod/github.com/studio-b12/[email protected]/client.go:164 +0x1a6 fp=0xc0001d98e8 sp=0xc0001d9808 pc=0x18d2476

Should parallel calls using single webdav client be supported?

Set GetBody on http.Request

A user of Kopia reported this error in kopia/kopia#1509:

Propfind "https://myhost.mydomain.com/kopia/p5f/7be/": http2: Transport: cannot retry err [http2: Transport received Serv
er's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

There appears to be an issue with some proxy servers that force connection closure after receiving the request and Go cannot automatically retry the request unless http.Request.GetBody is set.

Could this be added in gowebdav ?

Support for cookies

Hello Collaborators,

can we add the ability to set the CookieJar of the internal http.Client of the gowebdav.Client?
I got proprietary webdav server, which does authentication via kerberos. It is way faster to use the cookie after the initial authentication than to reauthenticate at every request.

I see 3 possibilities to implement this:

1. provide a SetJar method for the gowebdav.Client like this

   func (c *Client) SetJar(jar http.CookieJar) {
   	c.c.Jar = jar
   }

2. expose the internal http.Client by making the c field public or add an option to gowebdav.NewClient so a configured http.Client for the webdav client to use can be provided.

3. use the http.DefaultClient in gowebdav.NewClient function to initialize the c field.

I like number 2. with an option to provide a http.Client best, because it adds the most flexibility. Maybe there is a good reason why the library uses its own clean http.Client that I don't see right now. In that case I would like number 1 implemented.

Thoughts?

(I can provide the pull request if we reach a decision)

Stat() for directory does not return last modified time

Hello!

When I use Stat() for retrieve directory last modification time, I got zero-time.

In source code (client.go):

  if p.Type.Local == "collection" {
  if !strings.HasSuffix(f.path, "/") {
	  f.path += "/"
  }
  f.size = 0
  f.modified = time.Unix(0, 0) // Why ??
  f.isdir = true
  } else {
  f.size = parseInt64(&p.Size)
  f.modified = parseModified(&p.Modified)
  f.isdir = false
}

No support for Bearer Token authentication

Hello Collaborators,

Describe the bug
401 Authorization error on trying to connect to a WebDAV server with Bearer type of authentication

Software

• OS: Ubuntu 20.04 LTS
• Golang: go version go1.19.2 linux/amd64
• Version: latest studio-b12/gowebdav (loaded via go get)

To Reproduce

1. Connect to any WebDAV server which requires Bearer type of authentication. I used OwnCloud
2. Try to read the list of directories using c.ReadDir("/") (Any other operation also fails)
3. Error: 401 Authorization Error

Expected
No authorization error, the client should be able to connect to the server and execute any operation

Additional context
This seems to be due the fact that only two types of auth: Basic and Digest are supported.
I would be willing to work on a PR to add Bearer authentication support

improve concurrency

Here are my points which I have to think more about.

1. API changes in Authorize. I'd love to avoid it.
2. Losing the context (Client) in Authorize.
3. The Mutext, which is just needed for concurrency.

Originally posted by @chripo in #36 (comment)

Authentication fails when deployed to Windows machine without Go installed

I wrote an app on Windows using basic authentication, which appears as Go-http-client/2.0 on the server, if run from my developer machine with Go installed, and works well.

On a different machine without Go installed (or on my machine after uninstalling Go) the same client appears as Microsoft-WebDAV-MiniRedir/10.0.16299 on the server, and authentication fails.

I read quite a bit about dynamic linking if using the net package, but right now I am stuck.

codereview

hi contributors,
hi folks,

we improved the project in a nice way.
thank you for your work and interest.
most of your commits were picked without a pull requests, this was fine.
please see my modifications as a kind of talking to and with you by source code.
so feel free to review and to improve the source and project.

let's start to review these files

• requests.go
• client.go

404 isn't treated as expected

In Read/ReadStream no HTTP response code handling is done. The expected behaviour IMO is to promote 404 and similar errors to actual err responses.

Support range bytes request

I tried to call ReadStream with header SetHeader("Range", "bytes=10-100"), but it failed because ReadStream only accepts 200 http response code. The response code of range bytes request is 206 (Partial Content).

gowebdav/client.go

Line 332 in 3cd755d

if rs.StatusCode == 200 {

I think the above code needs to be revised as follows:
if rs.StatusCode == 200 || rs.StatusCode == 206 {

gowebdav library should not hush up errors

If we carefully see on following methods:

• client.mkcol(), client.Mkdir(), client.MkdirAll()
• client.copymove(), client.doCopyMove()
• client.Write() and client.WriteStream()

• it suppress real errors and returns only status code in the best case.

I propose to change this behaviour to allow user of our library to get original error message.

Can't upload file with content

Hello Collaborators,
during testing current version of gowebdav I notice that client's method WriteStream does not set request body even if body should be set for current request (for example during uploading non-empty file via gowebdav.exe -X PUT /my_folder/file.txt file.txt). Also, @chuckwagoncomputing face with this problem too, but he used propfind request instead.

Software

• OS: Windows 10 x64
• Golang: 1.10
• Version: latest commit in master is 008b27e

To Reproduce

1. build gowebdav/cmd/gowebdav with go build to get gowebdav.exe
2. set following environment's variables:
   1. ROOT -- WebDav Server's URL
   2. USER -- WebDav Server's username
   3. PASSWORD -- WebDav Server's password
3. suppose your WebDav Server's folder is my_folder, and your local file to upload located near gowebdav.exe, named file.txt and has some text (so actually file's size is not zero).
4. run following command: gowebdav.exe -X PUT /my_folder/file.txt file.txt
5. go to your WebDav Server via any other interface (putty, web browser etc), and you will see that file is exist but file's size is zero

Expected
Uploaded file contains body

Actually
Uploaded file does not contains body

memleak during multiple file upload when authentication is required

Hello Collaborators,

Describe the bug
A short description of what you think the bug is.

Software

• OS: Linux Mint 19 Tara
• Golang: go version go1.10.1 linux/amd64
• Version: latest studio-b12/gowebdav (loaded via go get)

To Reproduce
I've started to build a small tool which would allow file syncing via WebDAV. It's not fully ready but at least it allows to scan a folder and create a list of files which should be uploaded. The flow of the program now is the following:

1. Create list of files for uploading
2. Create N goroutines, each has its own gowebdav.Client
3. Filenames for uploading are sent to these goroutines via channels (fan-out), then each file is uploaded independently via Client.WriteStream. The file is closed via defer.
4. I used WebDAV API for Yandex.Disk, but I believe the same problem could be reproduced with any server with configured Basic authentication.

What I observe is enormous RAM usage by my tool (e.g. even if a folder contains files for 1G, it can easily take up to several gigabytes of RAM). I've tried to narrow the scope of the problem and here's what I've got:

• GODEBUG=gctrace=1 shows that upon each gc cycle it could collect 0 space, and the space constantly grows
• pprof.WriteHeapProfile at the end gives me a profile which shows that there's some inuse_space related to TeeReader from requests, though acquired by persistConn inside of http.Transport. It's not several gigabytes, but up to several hundred megabytes.
• I tried to use Client.SetTransport() providing my http.Transport with reduced IdleConnTimeout, MaxIdleConns, DisableKeepAlive and other similar things like explicit call . Nothing changed.
• Finally, I reverted gowebdav to revision prior to the fix for issue #14 - no additional memory has been taken (memory footprint was just about several megabytes according to gctrace)

I believe there's something wrong with current usage of TeeReader introduced in #b45378c. It looks like persistConn inside of a Transport somehow doesn't allow to collect the unused objects (i.e even if a new file gets uploaded, the data for the old one doesn't get cleared). Unfortunately I'm pretty new to Go and couldn't resolve the issue by myself.

Expected
Well, no memleak

Additional context
Just in case you're curious how do I use gowebdav exactly, here is the tool I'm using.

Please let me know if you need any additional info.

File remove problem

Some webdav servers (like yandex) send 204 status for successful delete operation. On client.go file (line: 195) you can add 204 status. Note: Thanks for this good client.

cmd: login / connect by pass-name [pass]

• parse default pass content (check)

pw

login: foo
url: https://host.tld/

• cmd args --pass pass-name

[err:WriteStream] file write problem: err:WriteStream

Hello Collaborators,

Describe the bug
A short description of what you think the bug is.

Software

• OS: win 11
• Golang: 1.18.1
• Version: 8

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected
A short description of what you expected to happen.

err:WriteStream xx: 500

Additional context
Add any other context about the problem here.

server is on WSL2 of docker, images is derkades/webdav.
I try to upload a image file

Apache HTTPS Server does not return `Dav` or `DAV` header on `OPTIONS` request

Here is the current version of Connect function:

// Connect connects to our dav server
func (c *Client) Connect() error {
	rs, err := c.options("/")
	if err != nil {
		return err
	}

	err = rs.Body.Close()
	if err != nil {
		return err
	}

	if rs.StatusCode != 200 || (rs.Header.Get("Dav") == "" && rs.Header.Get("DAV") == "") {
		return newPathError("Connect", c.root, rs.StatusCode)
	}

	return nil
}

but Apache HTTPS Server does not return Dav or DAV header, so application returns following message on any request:

Failed to connect due to: Connect https://mywebdav.me/: 200

Is it bug in library, or wrong settings of Apache HTTPS Server?

Use the tag Management release version to publish

gowebdav is generally used as a three-party librarie,and we prefer to use the release version rather than the unstable development version

Wrong content of uploaded file

Hello Collaborators,
During testing current version of gowebdav/cmd I notice that uploaded file has incorrect content if function getStream() can not find required file to open.

Software

• OS: Windows
• Golang: go version go1.11.4 windows/amd64
• Version: current version of gowebdav/cmd

To Reproduce

• call gowebdav.exe with args: -X PUT /webdav_folder/file.txt

Expected
File not found error returned

Actual
Application uploads file with it's path as content

Error on Folders with '+'

Hello Collaborators,

Describe the bug

When we want to Propfind a folder which contains folders with "+", the call fails. Due to the QueryUnescape at
https://github.com/studio-b12/gowebdav/blob/master/client.go#L141
the "+" results in a wrong foldername. in this case "lost+found" becomes "lost found".

Not sure what the purpose of QueryUnescape is, but once we remove it, it works just fine.

Software

• OS: linux
• Golang: 1.15
• Version: latest master

To Reproduce

1. Create a folder 'lost+found' in a folder 'a'
2. Propfind folder 'a'
3. Propfind returns 'lost found' as the folder name

Expected
Expected the folder name "lost+found" to stay that way

Www-Authenticate header's token should be treated as case-insensitive

Hi,

I've run into trouble using this library with a server that returns a Www-Authenticate header of "BASIC" instead of "Basic". This is allowed as per https://tools.ietf.org/html/rfc2617, but the check on Lines 46 and 48 of requests.go does a case-sensitive check against 'Digest' or 'Basic'.

Can this be changed to a case-insensitive match like
strings.Index(strings.ToLower(rs.Header.Get("Www-Authenticate")), "digest")?

Is there any workaround? I can't see any way to manually specify the authentication method.

Regards,

Wade

support Walk func

Webdav does not require the server to support recursive traversal, but recursive traversal is commonly used in some scenarios, so we should support the walk function for users. just like filepath.Walk()

400 when passing whitespace in path args to c.Copy()

Hello Collaborators,

Describe the bug

c.Copy() doesn't handle whitespace characters in file/folder names.

Software

• OS: ubuntu
• Golang: 1.15.7
• Version: latest

To Reproduce

1. Create a file or folder with whitespace in the name on the remote webdav server.
2. Make a request with client.Copy with oldpath and newpath being strings with spaces in the name.
3. 400 Error is returned

Expected
Spaces should be handled.

Additional context
Add any other context about the problem here.

nil pointer exception in req() method

Software

OS: Windows 10 x64
GO: go1.9.2 windows/amd64
gowebdav: latest version from master

go env

set GOARCH=386
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=D:\Go\
set GORACE=
set GOROOT=D:\Programs\Go
set GOTOOLDIR=D:\Programs\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set GO386=sse2
set CC=gcc
set GOGCCFLAGS=-m32 -fmessage-length=0 -fdebug-prefix-map=C:\Users\PC\AppData\Local\Temp\go-build886287271=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=0
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

Issue description

We use http.Do() method inside c.req(), and do not check for errors returned. So, it is possible to receive nil response, and got nil pointer dereference panic in line 42.
http.Do() method will return non-nil error in following cases:

1. Request is nil
2. Response missing Location header
3. Client failed to parse Location header
4. Method "request.GetBody()" returns error
5. Http.Client.Send() returns error
6. Client timeout was exceeded
   In all cases except of (5) this method returns nil as a poiner to http.Response, which will force app to panic later.

Conclusion

Taking into account the above mentioned, we need to perform error-checking for http.Do() method results.

[BUG/FEATURE] OneDrive Support

Hello Collaborators,

Describe the bug
When using OneDrive as the WebDav server things get weird. In short: The authentication fails because of Microsoft.

Software

• OS: Linux (Docker)
• Golang: 1.19.4
• Version: 60ec5ad

To Reproduce

1. Create a client using OneDrive credentials
2. Try to use any (authenticated) request (e.g. ReadDir())
3. Response is empty / unexpected (e.g. no files)

Expected
At least an authentication error should be returned. Even better would be support for MS's need to be proprietary.

Additional context
Microsoft (as expected) does not follow industry standards and instead of returning 401 on unauthenticated requests they return 302 to redirect you to the login page. The login page then return a 200 status so the webdav client thinks it is a valid result for the original request. On its own this would not be too bad as you could catch the redirect and handle it like a 401. But since Microsoft use their own authentication method (MS-PASS with Passport1.4) you also need a new authenticator.

Digest auth usage

Hello Collaborators,

From the docs I see there is a support to use DigestAuth, but I couldn't find how exactly am I supposed to pass the credentials for using Digest Auth, I see by default gowebdav uses basic auth. How can I explicitly ask it to use digest auth? Could you provide an example on how to use DigestAuth with gowebdav client?

Software

• OS: Mac
• Golang:

go version
go version go1.17.1 darwin/amd64

• Version:

github.com/studio-b12/gowebdav v0.0.0-20230203202212-3282f94193f2

Additional context
I'm trying to upload my HLS content to AWS Elemental MediaPackager

ReadDir omits returning the first result if it is a directory

Hello Collaborators,

I am reading a directory using ReadDir API. The contents of the directory are two directories. ReadDir only returns the second directory.

I debugged the code, and the culprit is the following block of code in client.go file

	if skipSelf {
		skipSelf = false
		if p := getProps(r, "200"); p != nil && p.Type.Local == "collection" {
			r.Props = nil
			return nil
		}
		return newPathError("ReadDir", path, 405)
	}

If I comment out this code, it works as expected. I do not know what this skipSelf thingy is. It does not seem to have it there be necessary when I used it

Software

• OS: Windows 10
• Golang:
• Version: 1.15

To Reproduce

1. ReadDir on a folder that has two folders (no files)
2. You will the second folder in the result

Expected
ReadDir should return both dirs.

Additional context
My test file: main.go - you need to add your WebDav Url and username/password.

package main

import (
"log"
"path/filepath"

"github.com/studio-b12/gowebdav"

)

func main() {
root := "xxxx"
user := "yyyy"
password := "zzzz"

c := gowebdav.NewClient(root, user, password)
err := c.Connect()
if err != nil {
	log.Fatalf("could not connect %v\n", err)
}

readDir(c, "/")

}

func readDir(c *gowebdav.Client, dpath string) {
fi, err := c.ReadDir(dpath)
if err != nil {
log.Fatalf("could not read dir %v\n", err)
}
for _, file := range fi {
if file.IsDir() {
d := filepath.ToSlash(filepath.Join(dpath, file.Name()))
log.Printf("Directory: %v\n", d)
readDir(c, d)
continue
}
log.Printf("File: %v\n", file.Name())
}
}