Experimental Kubernetes Device Plugin for Coral Edge TPU
kkohtaka / edgetpu-device-plugin Goto Github PK
View Code? Open in Web Editor NEWExperimental Kubernetes Device Plugin for Coral Edge TPU
License: Apache License 2.0
Experimental Kubernetes Device Plugin for Coral Edge TPU
License: Apache License 2.0
First of all thank you for sharing your work on this plugin, it has saved me a lot of time already.
I have started working on k3s compatibility for this device plugin and I have gotten to the point when the plugin discovers available TPUs and registers them with the kubelet.
$ kubectl describe nodes
...
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 100m (2%) 0 (0%)
memory 70Mi (7%) 170Mi (18%)
ephemeral-storage 0 (0%) 0 (0%)
kkohtaka.org/edgetpu 1 1
I0327 17:02:58.014104 1 plugin.go:98] Started gRPC service on plugin socket
I0327 17:02:58.014183 1 plugin.go:101] Started monitoring devices
I0327 17:02:58.014211 1 plugin.go:49] gRPC server started.
I0327 17:02:58.015416 1 plugin.go:118] Opened connection to kubelet socket
I0327 17:02:58.015486 1 plugin.go:121] Registering dpServer: &{[] 0xd58100 0xd58140}
I0327 17:02:58.017211 1 plugin.go:134] Registered device plugin
I0327 17:02:58.019309 1 server.go:56] Start watching devices
I0327 17:02:58.019419 1 server.go:66] Update a device list
I0327 17:02:58.019455 1 server.go:126] Starting Edge TPU device monitor
I0327 17:03:03.184672 1 server.go:155] Edge TPU became active.
I0327 17:03:03.185096 1 server.go:66] Update a device list
I0327 17:04:03.390627 1 server.go:79] Container TPU request: &ContainerAllocateRequest{DevicesIDs:[42],}
I0327 17:04:03.390765 1 server.go:80] Allocating devices... Device IDs: [42]
side note: it looks like the device id is hardcoded to 42 so only 1 TPU is currently allowed per node, do you plan to support multiple devices?
I am able to schedule a pod which requests a TPU but the container fails to start due to:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 86s default-scheduler Successfully assigned default/edgetpu-demo-54f5l to raspberrypi
Normal Pulling 2s (x2 over 84s) kubelet, raspberrypi pulling image "quay.io/kkohtaka/edgetpu-demo:arm32"
Warning Failed 2s kubelet, raspberrypi Error: failed to generate container "41e1245b846a2a815f54ca40741d10fc071f91b34eadf22309c52f949ea1d4ce" spec: failed to set devices mapping [&Device{ContainerPath:/dev/bus/usb,HostPath:/dev/bus/usb,Permissions:rw,}]: not a device node
Normal Pulled 1s (x2 over 2s) kubelet, raspberrypi Successfully pulled image "quay.io/kkohtaka/edgetpu-demo:arm32"
Warning Failed 1s kubelet, raspberrypi Error: failed to generate container "d72f48a0b04a560b1c7e81ec20d686b6ca3710f0a02fd38cecb1b937b52e8d05" spec: failed to set devices mapping [&Device{ContainerPath:/dev/bus/usb,HostPath:/dev/bus/usb,Permissions:rw,}]: not a device node
It looks like I need to specify an absolute path to the device but I'm not sure what that would be, could you point me in the right direction? I'm using a raspberrypiB+ with a coral accelerator usb for testing
If you're open to it, I'd be happy to submit a PR for k3s support once I get this working.
I'm running your device plugin in OpenShift 3.11 which has kubernetes under the hood. I realize you might not have done any testing with OCP but figured you might be able to help. Here is the setup:
I am able to connect the TPU to the physical host and run the python demo code to show it works. I can even assign the USB device to a compute node VM and in the VM run the python demo code to show that the VM sees the device and can talk to it.
Before I do anything with the daemonset I see this on the physical host:
$ lsusb
Bus 002 Device 005: ID 1a6e:089a Global Unichip Corp.
I then use your yaml to deploy the daemonset. One of the pods in the daemonset shows:
oc logs -f edgetpu-device-plugin-52sjt
I0812 16:17:10.264373 1 plugin.go:98] Started gRPC service on plugin socket
I0812 16:17:10.264399 1 plugin.go:101] Started monitoring devices
I0812 16:17:10.264404 1 plugin.go:49] gRPC server started.
I0812 16:17:10.264607 1 plugin.go:118] Opened connection to kubelet socket
I0812 16:17:10.268002 1 server.go:56] Start watching devices
I0812 16:17:10.268025 1 server.go:66] Update a device list
I0812 16:17:10.268092 1 plugin.go:132] Registered device plugin
I0812 16:17:15.369094 1 server.go:150] Edge TPU became active.
I0812 16:17:15.369137 1 server.go:66] Update a device list
So far that all looks good. I then deploy the sample with your yaml file and it comes back with:
oc logs -f edgetpu-demo-9cb92
ERROR: Failed to retrieve TPU context.
ERROR: Node number 0 (edgetpu-custom-op) failed to prepare.
Failed in Tensor allocation, status_code: 1
And then if I go back to the physical host:
lsusb
Bus 002 Device 006: ID 18d1:9302 Google Inc.
It changed from 002:005 to 002:006. It is like the physical host thinks the USB device was disconnected and reconnected. I have see this before I started using your code where I'd run a container on the VM and it would fail, see the device changed on the host, readd device to VM, and run container on VM again...and it works.
Would you have any insight into why talking to the device or somehow assigning it to a container causes this name change? Thank you.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.