wlandau/crew.aws.batch

A {crew} launcher plugin for AWS Batch

crew.aws.batch's Issues

GitHub interactions are temporarily limited because the maintainer is out of office.

Vacation mode

When this issue is open, vacation mode is turned on. That means Github interactions are temporarily limited, so users cannot open or comment on issues or discussions until I return and re-enable interactions (see return date below). When this issue is closed, vacation mode is turned off and interactions are re-enabled and possible again.

Thanks

Vacation mode helps me rest because it prevents tasks from piling up in my absence. Thank you for your patience and understanding.

Day of my return

Already returned.

Vacation mode source code

• https://github.com/wlandau/dotfiles/blob/main/github/vacation.R

Job monitoring utilities

It would be great to have standalone R function utilities to manage batch jobs. These would run in the user's interactive session outside the targets pipeline / crew controller. I am thinking of covering the same functionality as qsub, qstat, and qdel in SGE (sbatch, squeue, and scancel in SLURM), plus log files. Proposal:

1. crew_aws_batch_submit(): submit a job that runs some code (R or shell). This could help e.g. submit a targets pipeline as a Batch job which submits other Batch jobs.
2. crew_aws_batch_status(): get the status of jobs in a given job queue / job definition.
3. crew_aws_batch_terminate(): terminate one or more jobs with specific job names/IDs/ARNs.
4. crew_aws_batch_logs(): log files for one or more jobs, or for an entire job definition. This would really help detect tricky worker-level errors such as running out of memory or hitting a price spike that terminates spot instances.

Paginate logs from the monitor

Because of paws-r/paws#721, the log() method of the monitor currently does not use pagination. When that issue is fixed, I will replace this:

crew.aws.batch/R/crew_aws_batch_monitor.R

Lines 753 to 759 in cf88f67

	pages <- list( # TODO: paws.common::paginate() # nolint
	client$get_log_events(
	logGroupName = private$.log_group,
	logStreamName = log_stream_name,
	startFromHead = start_from_head
	)
	)

with this:

pages <- paws.common::paginate(
  client$get_log_events(
    logGroupName = private$.log_group,
    logStreamName = log_stream_name,
    startFromHead = start_from_head
  )
)

and test it with this:

crew.aws.batch/tests/monitor/test-jobs.R

Lines 87 to 129 in 06f92d5

	test_that("job logs", {
	x <- crew_aws_batch_monitor(
	job_definition = "crew-aws-batch-test",
	job_queue = "crew-aws-batch-job-queue",
	region = "us-east-2"
	)
	x$register(
	image = "alpine:latest",
	platform_capabilities = "EC2",
	memory_units = "gigabytes",
	memory = 1,
	cpus = 1,
	seconds_timeout = 600,
	tags = c("crew_aws_batch_1", "crew_aws_batch_2"),
	propagate_tags = TRUE
	)
	on.exit(x$deregister())
	job <- x$submit(
	command = c("echo", "done with container\ndone with job"),
	memory_units = "mebibytes",
	memory = 128,
	cpus = 1
	)
	attempts <- 0L
	done <- c("succeeded", "failed")
	while (!((status <- x$status(id = job$id)$status) %in% done)) {
	message(
	paste(
	"job status:",
	status,
	sample(c("-", "\\", "|", "/"), size = 1L)
	)
	)
	attempts <- attempts + 1L
	if (attempts > 60L) {
	stop("job did not finish")
	}
	Sys.sleep(5)
	}
	log <- x$log(id = job$id)
	expect_true(tibble::is_tibble(log))
	expect_equal(log$message, c("done with container", "done with job"))
	})

Expose the signal argument of the launcher constructor

C.f. wlandau/crew#141. Needs to wait until after crew > 0.7.0 is on CRAN.

R6 private objects/methods

Separate job definition management from the monitor

A "monitor" in the crew ecosystem should just list and terminate jobs, not launch jobs or configure job definitions. I have plans to create a new job definition class for this.

Job definition guidance

Either a function to create new job definitions or documentation for how to do this. Sketch using the CLI:

aws batch register-job-definition \
  --job-definition-name YOUR_JOB_DEFINITION_NAME \
  --type container \
  --container-properties "image=AWS_ACCOUNT.dkr.ecr.AWS_REGION.amazonaws.com/IMAGE_NAME:IMAGE_TAG, vcpus=4, memory=16000"