aurora-scheduler/scheduler

Maintenance fork of Apache Aurora's Scheduler

scheduler's Issues

Redirect agent links in the web GUI to common reverse proxy

I would like to have the agent pointing links in the web GUI point to a reverse proxy, so that I can look at the job logs without having to tunnel inside the agents cluster. Is that actually possible?

Upgrade Thrift dependency

We currently use Thrift 0.10.0. Current version is 0.13.0

There's been a bunch of improvements on the Java side and it would be worth while looking at picking them up. https://github.com/apache/thrift/blob/master/CHANGES.md

Upgrade Jcommander

We're currently on version 1.72. Latest version available is 1.78

Moving python client to its own repo

As mentioned on the Aurora slack channel a few days ago, I'm currently ill equipped in terms of bandwidth and knowledge to maintain the Aurora python client.

Coupled with the fact that Python 2 will no longer be seeing security updates, I feel it's the right time to move the python client to its own repository. Hopefully, this opens up the opportunity for someone else to take over maintenance on their own schedule and bring it to Python 3.

I will do my best to simultaneously contribute new features and or changes necessary to allow the aurora python client to maintain compatibility with the later versions of the scheduler.

Migrate classname

Move from org.apache.aurora to io.github.aurora-scheduler

This may be easy or it may not be worth the time.

Replace deprecated npm modules

When building aurora, we get the following warning messages:

npm WARN deprecated [email protected]: 🙌  Thanks for using Babel: we recommend using babel-preset-env now: please read https://babeljs.io/env to update!
npm WARN deprecated [email protected]: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
npm WARN deprecated [email protected]: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated

We should replace the deprecated modules.

Upgrade Gradle to latest 5.x version

It'd be nice to upgrade gradle to the latest 5.x version before releasing 0.23.0

CI does not work on pull_request

CI does not work on pull_request.
please take a look at https://github.com/aurora-scheduler/scheduler/pull/445/checks

Update and resolve npm issues

When building the vagrant setup, npm spits out all sorts of error messages, including some saying a dependency was deprecated and a recommendation to switch to a new dependency.

[NOTICE] Project will be archived on March 31st, 2023

I have moved to a new job a few months ago and I don't want to give the impression that this project is still alive or maintained.

Thanks to all for joining on this adventure. We have reached our final stop ✋🏼.

Actions Ubuntu Xenial env is gone

We either need to upgrade our builds to 18.04 or higher or stop the CI.

Upgrade Quartz scheduler

We're currently on 2.2.2, latest version is 2.3.0

Upgrade curator dependencies

Currently using 2.12.0, an upgrade to 4.3.0 would be great if possible.

Update Guava and/or Guice dependencies

Upgrade Guave and/or Guice to the latest version possible.

Upgrade Mesos dependency to 1.9.0

Bump up Mesos dependency to 1.9.0.

This is likely the last release of Mesos and will in all likelihood be the last release of the Aurora Scheduler.

Bintray is being sunset

JFrog recently announced that bintray is being sunset.

There's a build script that pulls down Mesos python eggs from bintray. Assuming this script is still used, we'd need to find some alternative to hosting those binaries before Bintray is turned off in May.

track the number of offers filtered out by scheduler plugins

Filtering out offers may cause starvation even we have enough offers.
In this case, we need keep track the number of offers that could not make to the final offer set.

Upgrade to JDK 11

It would be nice to upgrade the codebase's JDK compatibility to JDK 11.

JDK 15 has just been released and has new garbage collectors that could improve the scheduler's performance in some scenarios.

Upgrading to JDK 11 would be a stepping stone on the way to 15.

Scheduling failed due to Duplicate key HostOffer

In our clusters we observe that we got duplicated host offer key error and aurora is unable to proceed with other tasks.

W0513 00:15:35.981 [TaskGroupBatchWorker, TaskSchedulerImpl] Task scheduling unexpectedly failed, will be retried java.lang.IllegalStateException: Duplicate key HostOffer{offer=id {
  value: "704c1042-f056-4582-b0b4-30231ca4ce96-O11593022"
}
framework_id {
  value: "9f48d831-63e7-4556-86ab-463a69389e4d-0000"
}
agent_id {
  value: "704c1042-f056-4582-b0b4-30231ca4ce96-S1890"
}
hostname: "******"
resources {
  name: "ports"
  type: RANGES
  ranges {
    range {
      begin: 10000
      end: 10150
    }
  }
  role: "*"
}

, hostAttributes=IHostAttributes{host=*******, attributes=[IAttribute{name=hostname, values=[*****]}, IAttribute{name=az, values=[us-central1-b]}, IAttribute{name=dedicated, values=[test/onboard]}, IAttribute{name=host, values=[10.180.21.192]}, IAttribute{name=nodeID, values=[autoscaler-gp]}], mode=NONE, slaveId=704c1042-f056-4582-b0b4-30231ca4ce96-S1890}, nonZeroCpuAndMem=true}
            at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133)
            at java.util.HashMap.merge(HashMap.java:1254)
            at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320)
            at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
            at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
            at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
            at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
            at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
            at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
            at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
            at io.github.aurora.scheduler.offers.HttpOfferSetImpl.processResponse(HttpOfferSetImpl.java:334)
            at io.github.aurora.scheduler.offers.HttpOfferSetImpl.getOrdered(HttpOfferSetImpl.java:265)
            at org.apache.aurora.scheduler.offers.HostOffers.getAllMatching(HostOffers.java:176)
            at org.apache.aurora.scheduler.offers.OfferManagerImpl.getAllMatching(OfferManagerImpl.java:173)
            at org.apache.aurora.scheduler.scheduling.TaskAssignerImpl.lambda$findMatches$4(TaskAssignerImpl.java:234)
            at java.lang.Iterable.forEach(Iterable.java:75)
            at org.apache.aurora.scheduler.scheduling.TaskAssignerImpl.findMatches(TaskAssignerImpl.java:224)
            at org.apache.aurora.scheduler.scheduling.TaskAssignerImpl.maybeAssign(TaskAssignerImpl.java:260)
            at io.github.aurora.scheduler.scheduling.ProbabilisticPriorityAssigner.maybeAssign(ProbabilisticPriorityAssigner.java:105)
            at org.apache.aurora.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:83)
            at org.apache.aurora.scheduler.scheduling.TaskSchedulerImpl.scheduleTasks(TaskSchedulerImpl.java:154)
            at org.apache.aurora.scheduler.scheduling.TaskSchedulerImpl.schedule(TaskSchedulerImpl.java:108)
            at org.apache.aurora.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:83)
            at org.apache.aurora.scheduler.scheduling.TaskGroups$1.lambda$run$0(TaskGroups.java:174)
            at org.apache.aurora.scheduler.BatchWorker$Work.apply(BatchWorker.java:117)
            at org.apache.aurora.scheduler.BatchWorker.lambda$processBatch$3(BatchWorker.java:210)
            at org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult.apply(Storage.java:146)
            at org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult.apply(Storage.java:141)
            at org.apache.aurora.scheduler.storage.durability.DurableStorage.lambda$doInTransaction$0(DurableStorage.java:202)
            at org.apache.aurora.scheduler.storage.mem.MemStorage.write(MemStorage.java:96)
            at org.apache.aurora.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:83)
            at org.apache.aurora.scheduler.storage.durability.DurableStorage.doInTransaction(DurableStorage.java:201)
            at org.apache.aurora.scheduler.storage.durability.DurableStorage.write(DurableStorage.java:224)
            at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.write(CallOrderEnforcingStorage.java:132)
            at org.apache.aurora.scheduler.BatchWorker.processBatch(BatchWorker.java:207)
            at org.apache.aurora.scheduler.BatchWorker.run(BatchWorker.java:199)
            at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:66)
            at com.google.common.util.concurrent.Callables$4.run(Callables.java:119)
            at java.lang.Thread.run(Thread.java:748)

expected: aurora skip this error and try to schedule other tasks.

Update Zookeeper and Curator

We're currently running on a fairly old version of both of these libraries.

It would be nice to upgrade to newer versions that have a lot of bug fixes.

Tagging this as 0.24.0 but it may need to wait until 1.0.0.