Comments (3)
Hi, the per-batch job EBS provisioning code was retired in the latest version of aegea, and is not recommended because of hazards like the one you encountered. Note that after this failure to unmount, you will end up with an orphaned EBS volume that your organization will continue to pay for until you clean it up. This is quite hazardous if nobody is watching out for this type of error, because EBS volumes are not cheap.
I recommend using the EFS auto-mount functionality in the latest version of aegea instead. That way your jobs can use a shared EFS filesystem as their scratch space, and Batch automatically manages mounting and unmounting it for you.
If you must continue to use the EBS batch job volume code, it was updated to incorporate tracking down and terminating processes that prevent a clean unmount:
https://github.com/kislyuk/aegea/blob/develop/aegea/ebs.py#L180
You could try forking your version of aegea and updating this line to see if it makes a difference.
from aegea.
In addition to the line above, if you take a look at the commit that originally introduced it in v2.8.0, there is another change that makes the Batch cleanup handler perform a cd /
before trying to unmount. This is important because if the top level user process descended into a subdirectory in the mount, it will hold an advisory lock on that subdirectory and the mount will be seen as busy.
from aegea.
Hello,
Thanks again for all of this advice. I was going to just keep an eye on the EBS volumes to make sure there weren't any orphaned ones, but after this message I decided to just update to aegea 4.0.1 and just make the dependencies work out. So far so good.
Thanks again for your quick and helpful reply- much appreciated.
Best,
MO
from aegea.
Related Issues (20)
- How does aegea launch batch jobs? HOT 7
- Launch command stuck when launching instances in a private subnet HOT 1
- Initialize on-demand and don't fail on read-only FS
- ssh: integrate with vscode remote dev
- Provide friendly install instructions for awscli
- python 3.6 compat
- Retrieve and display cloud-init log on build-ami failure
- Make config parameters consistent with key=value argparse parameters
- Migrate from github3 to gh cli
- Manage SSH config to make "ssh aegea-HOST" equivalent to "aegea ssh HOST"
- launch: allow launch with PowerUser and iam:PassRole only HOT 1
- IMDSv2 compatibility
- aegea launch will overwrite role definition HOT 2
- Cloudtrail reader
- Use spot placement score API
- add support for amazon-linux-2022 ami HOT 2
- Migrate to ec2instanceconnectcli
- NVMe volume mounts do not persist upon reboot HOT 2
- Enable ECS-EBS integration
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aegea.