juliacloud/awstools.jl

awstools.jl's Issues

recursive download/copy from S3Path

This one has gotten me before and It would be good if download(srcdir, destdir; recursive=true) and copy(srcdir, destdir; recursive=true) worked to copy the folder structure.

The workaround right now is to use sync. but that wastes time (and potentially causes problems), but uploading contents of the destination to the source.

For backrun evaluation I find myself wanting to merge multiple remote directories into a single local directory,
so I can combine data from multiple 6 month runs to analyse.

I know {} and I have both complained about this on slack before, but apparently neither of use openned an issue to track it.

Remove _s3_list_objects once s3_list_objects in AWSS3 can replace it

[Copied over from GitLab]

See JuliaCloud/AWSS3.jl#34
We might have to wait for the 1.0 upgrade for this as well

Allow syncing only of files that match some rule

Often you only want to download files matching a particular rule,
e.g. only files with a particular filename.

I was thinking we could make sync take an extra argument which is a function
that maps from a path on the source (i.e. if downloading that would be the S3Path) to a boolean as to if it should be syncronised.

Right now as a hack I have the code to only download "financial_data.jlso"

using FilePathsBase
using AWSTools.S3

# The following two lines might not be needed
FilePathsBase.expanduser(p::S3Path)=p
FilePathsBase.isabs(::S3Path)=true

rootdir= S3Path("s3://eis-jobresultsbucket-12zhfo0p1o5ro/backrun/")
local_rootdir = p"mess_bucket/"
@sync for rpath in S3.list_files(rootdir)
    @async if basename(rpath)=="financial_data.jlso"
        local_rel_dir = dirname(String(relative(rpath, rootdir)))
        local_dir = Path(mkpath(joinpath(String(local_rootdir), local_rel_dir)))
        sync(rpath, joinpath(local_dir, "financial_data.jlso"))
    end
end 

But if we added this as a feature of sync then we could be much simpler:

sync(fn -> basename(fn) == "financial_data.jlso", rpath, lpath)

Online Tests: file fails to upload

Upload to S3: Test Failed at /eph/builds/8NJsKxdh/0/JuliaCloud/AWSTools.jl/test/S3.jl:239
  Expression: exists(dest)
Stacktrace:
 [1] (::var"#65#68"{S3Path})(::String, ::IOStream) at /eph/builds/8NJsKxdh/0/JuliaCloud/AWSTools.jl/test/S3.jl:239
 [2] mktemp(::var"#65#68"{S3Path}, ::String) at ./file.jl:611
 [3] mktemp(::Function) at ./file.jl:609
 [4] top-level scope at /eph/builds/8NJsKxdh/0/JuliaCloud/AWSTools.jl/test/S3.jl:230
 [5] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
 [6] top-level scope at /eph/builds/8NJsKxdh/0/JuliaCloud/AWSTools.jl/test/S3.jl:227
 [7] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
 [8] top-level scope at /eph/builds/8NJsKxdh/0/JuliaCloud/AWSTools.jl/test/S3.jl:212
 [9] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
 [10] top-level scope at /eph/builds/8NJsKxdh/0/JuliaCloud/AWSTools.jl/test/S3.jl:81
 [11] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
 [12] top-level scope at /eph/builds/8NJsKxdh/0/JuliaCloud/AWSTools.jl/test/S3.jl:71

A couple other tests also fail checking for the existence of the uploaded file.

Online Tests: Sync modified dest file failed

[ Info: Running ONLINE S3 tests
┌ Warning: `sync(src::AbstractString, dest::AbstractString; delete::Bool=false, config::AWSConfig=aws_config())` is deprecated, use `sync(Path(src), Path(dest); delete=delete)` instead.
│   caller = macro expansion at S3.jl:363 [inlined]
└ @ Core /mnt/builds/YryZfyyh/0/JuliaCloud/AWSTools.jl/test/S3.jl:363
Sync modified dest file: Test Failed at /mnt/builds/YryZfyyh/0/JuliaCloud/AWSTools.jl/test/S3.jl:399
  Expression: read(dest_files[1], String) == (s3_objects[3])["Content"]
   Evaluated: "Modified in dest." == "Hello World!"
Stacktrace:
 [1] macro expansion at /mnt/builds/YryZfyyh/0/JuliaCloud/AWSTools.jl/test/S3.jl:399 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [3] macro expansion at /mnt/builds/YryZfyyh/0/JuliaCloud/AWSTools.jl/test/S3.jl:386 [inlined]
 [4] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [5] macro expansion at /mnt/builds/YryZfyyh/0/JuliaCloud/AWSTools.jl/test/S3.jl:313 [inlined]
 [6] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [7] macro expansion at /mnt/builds/YryZfyyh/0/JuliaCloud/AWSTools.jl/test/S3.jl:212 [inlined]
 [8] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [9] macro expansion at /mnt/builds/YryZfyyh/0/JuliaCloud/AWSTools.jl/test/S3.jl:81 [inlined]
 [10] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [11] top-level scope at /mnt/builds/YryZfyyh/0/JuliaCloud/AWSTools.jl/test/S3.jl:71

Switch from AWSCore to AWS

Similar to the change in https://github.com/JuliaCloud/AWSS3.jl/pull/107/files, it would be nice to have this package compatible with the newer versions of things. Specifically around use of the legitimate AWS.AWSConfig type rather than the proxy AWSCore.AWSConfig.

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Remove `S3` submodule

Involves:

• move presign to AWSS3 (#26)
• remove the module, including any code/deprecations of S3Path, sync, presign i.e. delete
• tag a breaking release (v2.0)

Move `presign` to AWSS3

Move presign to AWSS3. Moving test may take a bit of work :)

Online Tests: Sync S3 directories failed

 [ Info: Running ONLINE S3 tests
 ┌ Warning: `sync(src::AbstractString, dest::AbstractString; delete::Bool=false, config::AWSConfig=aws_config())` is deprecated, use `sync(Path(src), Path(dest); delete=delete)` instead.
 │   caller = macro expansion at S3.jl:363 [inlined]
 └ @ Core /mnt/builds/KQ18Jxiv/0/JuliaCloud/AWSTools.jl/test/S3.jl:363
Sync S3 directories: Error During Test at /mnt/builds/KQ18Jxiv/0/JuliaCloud/AWSTools.jl/test/S3.jl:40
   Test threw exception
   Expression: size(dest_file) == size(src_file)
   404 -- AWSException
   HTTP.ExceptionRequest.StatusError(404, "HEAD", "/AWSTools.jl/04036ad3-03b0-4aa5-ac8b-339f0fdd7f7e/folder2/folder/file3", HTTP.Messages.Response:
   """
   HTTP/1.1 404 Not Found
   x-amz-request-id: {}
   x-amz-id-2: LOT/{}
   Content-Type: application/xml
   Transfer-Encoding: chunked
   Date: Sun, 09 Feb 2020 02:22:16 GMT
   Server: AmazonS3
   
   """)

Online S3 tests, no `ArgumentError` thrown

Sync non-existent directory: Test Failed at /mnt/builds/2gsuxSAE/0/JuliaCloud/AWSTools.jl/test/S3.jl:492
   Expression: sync(src_dir, dest_dir; delete=true)
     Expected: ArgumentError
   No exception thrown

Package compats for AWSS3 is wrong

The compat for AWSS3 in the Project.toml made by CompatHelper in #5 is wrong.

Firstly, it doesn't include 0.7 (and seems like it should) and second, it actually doesn't support 0.6.

(@v1.6) pkg> add AWSTools@1.10
...

julia> using Pkg; Pkg.pin("AWSTools")
...

# Not compatible with 0.7
(@v1.6) pkg> add AWSS3@0.7
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package AWSS3 [1c724243]:
 AWSS3 [1c724243] log:
 ├─possible versions are: 0.4.0-0.8.6 or uninstalled
 ├─restricted to versions 0.7 by an explicit requirement, leaving only versions 0.7.0-0.7.7
 └─restricted by compatibility requirements with AWSTools [83bcdc74] to versions: [0.6.0-0.6.13, 0.8.0-0.8.6] — no versions left
   └─AWSTools [83bcdc74] log:
     ├─possible versions are: 0.0.1-1.10.0 or uninstalled
     └─restricted to versions 1.10.0 by an explicit requirement, leaving only versions 1.10.0

# Supposed to be compatible with 0.6 but isn't
(jl_a8Kxbx) pkg> add AWSS3@0.6
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package FilePathsBase [48062228]:
 FilePathsBase [48062228] log:
 ├─possible versions are: 0.2.0-0.9.10 or uninstalled
 ├─restricted by compatibility requirements with AWSTools [83bcdc74] to versions: 0.9.0-0.9.10
 │ └─AWSTools [83bcdc74] log:
 │   ├─possible versions are: 0.0.1-1.10.0 or uninstalled
 │   └─restricted to versions 1.10.0 by an explicit requirement, leaving only versions 1.10.0
 └─restricted by compatibility requirements with AWSS3 [1c724243] to versions: 0.6.0-0.8.0 — no versions left
   └─AWSS3 [1c724243] log:
     ├─possible versions are: 0.4.0-0.8.6 or uninstalled
     └─restricted to versions 0.6 by an explicit requirement, leaving only versions 0.6.0-0.6.13

# Only compatible with 0.8
(jl_a8Kxbx) pkg> add AWSS3@0.8
   Resolving package versions...
    Updating `/private/var/folders/l2/zl6x61yd57q95ys8nvkfpmk80000gq/T/jl_a8Kxbx/Project.toml`
  [1c724243] + AWSS3 v0.8.6
  No Changes to `/private/var/folders/l2/zl6x61yd57q95ys8nvkfpmk80000gq/T/jl_a8Kxbx/Manifest.toml`

Expand Docker utilities

• Add Docker.tag (https://gitlab.invenia.ca/invenia/AWSClusterManagers.jl/merge_requests/70/diffs#note_81740)
• Support --build-arg in Docker.build (https://gitlab.invenia.ca/invenia/AWSClusterManagers.jl/merge_requests/70/diffs#note_81740)