Comments (14)
Holy smokes! As an extra precaution, can we avoid crossing file-system boundaries when we delete /tmp/gcs/.../ ?
from opengcs.
@jstarks I agree. If for whatever reason the unmounts fails, we need to be sure the content does not get erased by some code further down the path. We have been hurt very hard by a similar issue at some point with Docker for Windows, and really, you don't want to have to patch this kind of bug while in production.
from opengcs.
Do you have a proposed mitigation? I agree there is a risk here for sure when unmount fails but I don't know how I can cleanup the container in this case. Any deletes to the rootfs will also delete the mounted folder
from opengcs.
with unix fs semantics, I would assume that mv <mountpoint> /to-not-remove
might work... In the case of docker for Windows we decided that if an unmount fails, we don't remove the mount points at all.
from opengcs.
Reopening for further discussion.
After unmounting, you might be able to do a rename(rootfs, rootfs-removing) or something to make sure you're not crossing file system boundaries and then delete it.
from opengcs.
Would it just be easier to put it back at /binds/containerID/... instead? The location is under the control of docker, so it's a simple fix. At least that would guarantee not to be cleaned by GCS.
from opengcs.
I think the issue is that the gcs does the equivalent of rm -rf rootfs
which will delete everything including layer + plan9 data if the previous unmounts did not succeed. It doesn't matter where the mounts sources are located, because they eventually have to be mounted inside rootfs
so that the container can access them.
Right now the current cleanup code tries to unmount the mapped disks, the mapped directories and the layers and then does rm -rf rootfs
. If any of the unmounts fails, then it just logs it and continues forward, which can corrupt data on the delete.
It makes sense to me to have the same behavior as Docker for Windows where we simply don't clean up if the unmounts fail. Otherwise, we could corrupt the layers.
from opengcs.
Out of curiosity (and showing my ignorance to the finer implementation details here), why do you attempt to clean up in the first place? If the container goes away, the VM get's destroyed and then can't the layer just be removed on the host?
from opengcs.
Unmounting is necessary, because if the share isn't unmounted not all file operations on it are guaranteed to be flushed to the host. As for actually deleting the layers, I think this is less relevant now, but could be useful if attempting to run multiple containers in the same UVM. Otherwise a container would leak its private storage on exit.
from opengcs.
And, yeah, @simonferquel @gupta-ak I think it makes sense to emulate docker for Windows's behavior of not deleting layers which were not unmounted.
from opengcs.
@beweedon @jhowardmsft are you making progress on this? It seems like this on the critical path for LCOW, no?
from opengcs.
@jterry75, does the behavior of only deleting directories which we successfully unmounted seem good? We can make those changes if so.
Also, FYI to everyone else, Justin has been taking over the GCS from me, and at this point I won't really be doing any more work on it. I'll still be around to take part in discussions and stuff though :)
from opengcs.
I think we can skip the delete call on the layers if the unmount fails. The original issue was that we were not calling unmount for mapped directories so we are really talking about an unlikely failure path here. As far as I see it it only has two consequences to not delete the layers on failure to unmount:
- The UVM storage size will grow over time if this happens a lot. (Highly unlikely)
- The UVM will not be able to use that containerID again. (Unclear how likely this would be. I assume Docker creates a new ID per instance so we probably fine with this too even in a world where we have multiple containers per uvm) (If this becomes a problem we can always map external id's to unique gcs id's)
Everyone ok with me only deleting the layer if we have no unmount failures?
from opengcs.
Sounds good to me!
from opengcs.
Related Issues (20)
- execProcess should handle v2 external process
- V2 should support hot add/remove of NIC
- V2 exec process should consider UVMId
- Shutdown of UVM should send response to HCS
- bridge should make sure kernel version is 4.14.23 or greater
- In stress test, failed to get SCSI device name HOT 2
- [feature request] add version parameter HOT 2
- Error on run container with transparent network HOT 2
- Unspecified Error (0X80004005) and Failed to connect NIC
- init does too little and too much
- File ULIMITs HOT 1
- Intermittent DNS failures when running Alpine containers in user-defined docker-compose network
- vet error: set declared but not used HOT 1
- some "docker exec" commands fail with a cryptic error HOT 3
- LCOW: Docker Volumes don't support some fs operations that bind mounts support - CreateFile access denied HOT 2
- json.Unmarshal is incorrectly used
- json.Unmarshal is incorrectly used
- I would like to ask for some wrong answers HOT 4
- OT: Where to find tools/information about Hyper-V root on Linux
- Who owns C:\Program Files\Linux Containers? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opengcs.