juliacloud/awsbatch.jl

awsbatch.jl's Issues

AWSBatch Interface

Currently the AWSBatch interface is somewhere in between a high-level Julia interface and a low-level AWS API interface. We should do some iterating on this design to find something that that allows us to easily update with AWS API changes and new features while keeping things easy to use on the Julia side. Eventually we want the high-level Julia interface to look something like:

job = @batch memory=2048 begin
    using AWSClusterManagers
    addprocs(AWSBatchManager(5))
    ...
end

For inspiration look at AWSLambda.jl which has a @lambda and @lambda_eval for executing Julia code on Lambda.

Keeping this in mind I think we should adjust the current AWSBatch in the following way:

Make the BatchJob only keep track of the job_id. This would result in an interface change something like:

submit(; kwargs...) -> BatchJob
wait(::BatchJob) -> Bool
status(::BatchJob) -> JobState
describe(::BatchJob)

Applying these kind of changes to the job definition and other types would result in:

• The structures no longer needing to be mutable
• Aligning the low-level interface closer to the AWS API

Extract `@__VERSION__`

The @__VERSION__ macro should live outside of this package. I'm hoping that this can be added to Pkg.jl and Compat.jl later.

AWSBatch.jl/src/version.jl

Line 20 in 758025e

macro __VERSION__()

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Allow container environment override

We often override enviroment variables.
In particular

    --container-overrides environment="[{name=RUN_SCRIPT,value=$backrun_script_s3path}]"

but runbach does not currently take an enviroment kwarg (it takes kwarg for other things we can override)

AWSDocs: https://docs.aws.amazon.com/batch/latest/APIReference/API_ContainerOverrides.html#Batch-Type-ContainerOverrides-environment

Remove dependency on AWSSDK

AWSSDK has enormous amounts of code and is known to increase compile/load time. AWSCore or a dedicated service package (e.g., AWSTools or AWSS3 for S3) should be used instead.

Setup hosted documentation

Attempted to do so in #2 , setup the Deploy Key and Secret, doesn't to ever write.

Use Git_jll

Instead of using the command-line git in the tests we should switch to Git_jll as the command-line git may not be installed. The command would look something like:

using Git_jll
git() do git_path
    read(`$git_path ls-remote --tags https://github.com/JuliaLang/julia`, String)
end

Unfortunately there seems to be an issues at the moment:

• https://github.com/JuliaBinaryWrappers/Git_jll.jl/issues/2
• JuliaPackaging/Yggdrasil#1299

Migrate to AWS.jl

Currently this package uses AWSCore.jl and AWSSDK.jl, we should really migrate this to using AWS.jl.
We could possibly even drop the dependency on AWSTools.jl as well.

Instances of usage:

• AWS Batch
• AWS Cloudwatch
• AWS Configurations and Exceptions