hollance/yolo-coreml-mpsnngraph

Tiny YOLO for iOS implemented using CoreML but also using the new MPS graph API.

yolo-coreml-mpsnngraph's Issues

Question regarding semaphore concept and frame dropping

Hi,

I noticed you use the following concept in most of your projects:

let semaphore = DispatchSemaphore(value: 1)

extension ViewController: VideoCaptureDelegate {
  func videoCapture(_ capture: VideoCapture, didCaptureVideoFrame pixelBuffer: CVPixelBuffer?, timestamp: CMTime) {
    
      semaphore.wait()

      runCoreMLRequest(on: pixelBuffer) {
           semaphore.signal()
      }
}

You do this with the intention of dropping frames and this works, because you set AVCaptureVideoDataOutput.alwaysDiscardsLateVideoFrames = true.

However - and here comes my question - since DispatchSemaphore is a FIFO queue, won't you start the next CoreML request with the --oldest-- frame instead of the latest? I suspect that this is because videocaptures drops all newer frames. After the semaphore get's free, the frame that got emitted as the first one after the CoreML request started will be taken for the next request. Maybe here an exemplary sequence explaining what I mean:

frame 1 arrives
CoreML request starts with frame 1
frame 2 arrives and gets stuck in wait()
frame 3 gets dropped because previous frame processing stuck
frame 4 gets dropped because previous frame processing stuck
CoreML request finishes and calls semaphore.signal()
the frame that is stuck in wait() (frame 2) gets unblocked and runCoreMLRequest is started
CoreML request starts with frame 2

This would introduce an unnecessary delay of the duration of a signal NN forward pass.
Is that correct?

Reduced FPS

Thanks for much for doing this. I tried out the CoreML version of this and noticed that FPS for prediction is lower than what it used to be with bare bone MPS [using your Forge project]. Would you know why that might be the case ?

Support for yolo v3

Is there some effort in pipeline for conversion to YOLO v3?

Increasing the FPS

Thank you so much for your very inspiring work here. I am curious as to what parameter controls the fps at which the app runs at. I understand a higher fps requires more processing power but doesn't an iPhone have the processing power we need to crank up the fps even further?
This isn't so much an issue as it is a concept question. Thanks

Too many weights?

when converting my weights file to h5, I get the following:

Read 15867885 of 50676062.0 from Darknet weights.
Traceback (most recent call last):
  File "yad2k.py", line 271, in <module>
    _main(parser.parse_args())
  File "yad2k.py", line 263, in _main
    print('Warning: {} unused weights'.format(len(remaining_weights)))
TypeError: object of type 'float' has no len()

When using the downloaded weights-file it works as all the weights are read:

Saved Keras model to model_data/tiny-yolo-voc.h5
Read 15867885 of 15867885.0 from Darknet weights.
Saved model plot to model_data/tiny-yolo-voc.png

What am I doing wrong?

FPS go down when app runs a few minutes

on my iPhone X, the FPS from the demo app (coreml) starts at around 22/23 fps but goes down slowly but steadily - after ~ 2 minutes to about 13 fps - without moving the phone, just laying on the table.

YOLOv3, tiny-YOLOv3, tiny-YOLOv2 weird bounding boxes on Iphone X

I'm having trouble with recognition of objects on iPhone X (and iPhone 6s as I tested it) using the YOLOv3-CoreML and YOLOv2-Coreml. Not changing anything in the code gives weird results as you can see in the picture below:

Using the predict() function, it appears that iPhone X recognizes objects like the umbrella and others with 100% confidence and depicts bounding boxes as stripes. Changing confidenceThreshold and iouThreshold doesn't give any effect, as well as changing maxBoundingBoxes to 1-5. Using predictUsingVision does not produce any prediction at all on the screen.

The same problem I get with tiny-YOLOv3, YOLOv2 with custom and original models from Darknet using iPhone X.

However, using the same code on iPhone 6 (not s) produce an absolutely opposite result. It can predict objects successfully (using predict() function) with my custom model and original Darknet models:

I assumed that there is a problem with GPU changes Apple did start with iPhone 6s devices, but I haven't found any information about that. Does anybody have an issue like this? Did anybody try using the code on iPhone X device? I explicitly decided to test the code in the repository without any changes to provide a demonstration, the problem isn't about YOLOv3 or tiny-YOLOv3 or even YOLOv2 - the result is the same for me using any YOLO version.

You can find videos demonstrating the work on iPhone 6 and iPhone X:

iPhone X: https://imgur.com/a/EZOpr1W

iPhone 6: https://imgur.com/a/qQZtJAd

I will appreciate any help and assumptions you can provide, but as far as I tried to solve the problem for two weeks I didn't get the result, I wonder if there a positive cases using the code on iPhone X by other people. I tested out using different kinds of YOLO models but was always using the same code basis provided by hollance.

Will vision Framework able to identify the same object that it identify earlier?

For example, Vision found a honda car with blue color. Is it able to identify the same car that it previously "track" 3 minutes later? or the tracking is release from memory once it is out of the capture area. Basically, I want to know if a new car coming in, if it is the same car that it is previously track?

Error reading protobuf spec. validator error: Input confidence must have dataType DOUBLE

Hello, thanks for the great repo. I'm exporting a YOLOv3-tiny model from PyTorch > ONNX > CoreML using the instructions from your article.

The correctly-compiled mlmodel is here:
https://storage.googleapis.com/ultralytics/yolov3-tiny-float32.mlmodel

All successful except that my mlmodel outputs Float32, and on attempting to tie its outputs to the NMS inputs I get the following error.

/Users/glennjocher/PycharmProjects/onnx-coreml/venv/lib/python2.7/site-packages/coremltools/models/model.py:109: RuntimeWarning: You will not be able to run predict() on this Core ML model.Underlying exception message was: Error compiling model: "Error reading protobuf spec. validator error: Input confidence must have dataType DOUBLE".
  RuntimeWarning)

Casting the output to torch.float64 in PyTorch produces an error on the CoreML conversion unfortunately (NotImplementedError: Unsupported ONNX ops of type: Cast), so the conversion needs to be done after the existing mlmodel (but before NMS) in CoreML somehow. What's the easiest way to do this? Thanks! (BTW, I did all the box decoding in PyTorch natively, its only NMS I need to pipeline).

No Results Shown with Vision API - beta 3

Hi, for starters thank you for the great tutorials!

I cloned the last version of the repo that works with Vision on beta3, but when i run it on the device, there are no bounding box nor the elapsed time is shown. The app just shows the video feed.

Running with breakpoints, I noticed that observations and features are always 0, and

 let boundingBoxes = yolo.computeBoundingBoxes(features: features)
 let elapsed = CACurrentMediaTime() - startTime
 showOnMainThread(boundingBoxes, elapsed)

are never called.

The device is an iPhone 7.

EDIT: I forget to update to iOS beta 3, my bad!

I would like to improve detection performance

Hi guys,

I use Yolo for localization object on image. But when a person is on the side Yolo don't detect as a person..
No detection.. how to perform that ?

The version of Yolo use is V1 ? How to use the new version fo Yolo (V3) ?

Thank you for your answer.

throws error - 'float' has no len()

Error Message: Traceback (most recent call last):
File "yad2k.py", line 271, in
_main(parser.parse_args())
File "yad2k.py", line 263, in _main
print('Warning: {} unused weights'.format(len(remaining_weights)))
TypeError: object of type 'float' has no len()

Possible correction/suggestion:
In line 263 replace,

print('Warning: {} unused weights'.format(len(remaining_weights)))

with
print('Warning: {} unused weights'.format(remaining_weights))

Lower input sizes issues

  public static let inputWidth = 416
  public static let inputHeight = 416

I tried to set smaller values

  public static let inputWidth = 256
  public static let inputHeight = 256

but the rects aren't drawn correctly

re: Neural Engine vs. GPU

Huge fan of machinethink.net. Sorry to ask you this here, but wasn't sure of the best way to do so - you'd recently mentioned that you noticed CoreML inference falls back to GPU if there's a custom layer in the model; how did you know if it was deployed to the Neural Engine in the first place?

also, do you know if Metal Performance Shaders can access the Neural Engine? I ask because it seems that the MPS Graph API, which now supports both training and inference through a very Tensorflow-like fashion, seems to be the most low-level, robust abstraction Apple offers for custom ML, and it's quite bizarre that it appears so strictly bound to the GPU. It makes sense in that exists in Metal, but you'd really think that Apple would offer a computational graph API that sits above both BNNS (for CPU) and MPS (for GPU) and whatever specific instructions the Neural Engine supports, thus allowing you to express a model, train it, and run it across all three architectures depending on power and speed targets. It appears that CoreML now supports this (which is wondering how you knew the model was specifically running on the Neural Engine) but I find to be bizarre that if you actually decide to express and train a complex model in MPS, which is the only way to do it at the moment (and what CreateML is built off of) that you're then stuck, in a sense, to the GPU. I feel like I'm missing something here.

I recommend you use conda to create environments rather than pip / virtual env

there's no bin directory created
you can activate system wide environments
like python3 tensorflow
https://gist.github.com/johndpope/187b0dd996d16152ace2f842d43e3990

No bounding-box is detected on iphone6

Hi, hollance,
Thanks a lot for providing this great example. I am trying to reproduce on my iphone. This is my procedure below:

1. I tested the keras model, it produces the bbox as expected
2. I converted the keras model to coreML and put the coreML model in the project, when I launch the the app on iphone, there is no bbox is detected, then I printed the size of the predictions for each frame, it is 0. The confidence score is just very low around 10e-15, I guess it is filtered by the threshold.

I used your coreML model from github directly as well, the same results. Could you please provide some suggestions.

Thanks

increase numbers of label

Currently, I think the number of label is limited to 20 (in helper.swift) is it?

I tested it out with yolo9000 and it still limit only to above 20 array. How to increase the number of label to more than 20?

Will this work for OSX?

Hi, I want to implement this for OSX as well. But I am not sure if it will work.

Question: Anchors array

Hey, first of all thanks for amazing explanation and source code, really helpful to understand the workflow of such applications!

I could not find any explanation in your related blog post or in code to this variable:
let anchors: [Float] = [1.08, 1.19, 3.42, 4.41, 6.63, 11.38, 9.42, 5.11, 16.62, 10.52]

Where are these numbers come from ?
Thanks in advance!

Convert yolo failed using your way

Hi

I meet a problem when I convert yolo.weight (not tinyyolo) and hope you can help me.

Environment
Python 3.5.2 + Python 2.7.13

pip2 list

coremltools (0.4.0)
funcsigs (1.0.2)
h5py (2.7.0)
Keras (2.0.4)
mock (2.0.0)
numpy (1.13.1)
pbr (3.1.1)
pip (9.0.1)
protobuf (3.3.0)
PyYAML (3.12)
scipy (0.19.1)
setuptools (32.1.0)
six (1.10.0)
tensorflow (1.0.1)
Theano (0.9.0)
wheel (0.29.0)

pip3 list

h5py (2.7.0)
Keras (2.0.4)
numpy (1.13.1)
olefile (0.44)
Pillow (4.2.1)
pip (8.1.2)
protobuf (3.3.0)
pydot-ng (1.0.0)
pyparsing (2.2.0)
PyYAML (3.12)
scipy (0.19.1)
setuptools (25.2.0)
six (1.10.0)
tensorflow (1.0.1)
Theano (0.9.0)
wheel (0.29.0)

Step 1 Convert yolo.weights to yolo.h5 success. And I've tested the .h5 can work normally.

Step 2 Convert yolo.h5 to coreml
command: python2 coreml_yolo.py
output:

Traceback (most recent call last):
  File "coreml_yolo.py", line 12, in <module>
    coreml_model = coremltools.converters.keras.convert('yad2k/model_data/yolo.h5')
  File "/usr/local/lib/python2.7/site-packages/coremltools/converters/keras/_keras_converter.py", line 477, in convert
    predicted_feature_name = predicted_feature_name)
  File "/usr/local/lib/python2.7/site-packages/coremltools/converters/keras/_keras2_converter.py", line 149, in _convert
    model = _keras.models.load_model(model)
  File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 240, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 304, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/usr/local/lib/python2.7/site-packages/keras/layers/__init__.py", line 54, in deserialize
    printable_module_name='layer')
  File "/usr/local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 140, in deserialize_keras_object
    list(custom_objects.items())))
  File "/usr/local/lib/python2.7/site-packages/keras/engine/topology.py", line 2416, in from_config
    process_layer(layer_data)
  File "/usr/local/lib/python2.7/site-packages/keras/engine/topology.py", line 2385, in process_layer
    custom_objects=custom_objects)
  File "/usr/local/lib/python2.7/site-packages/keras/layers/__init__.py", line 54, in deserialize
    printable_module_name='layer')
  File "/usr/local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 140, in deserialize_keras_object
    list(custom_objects.items())))
  File "/usr/local/lib/python2.7/site-packages/keras/layers/core.py", line 706, in from_config
    function = func_load(config['function'], globs=globs)
  File "/usr/local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 198, in func_load
    code = marshal.loads(code.encode('raw_unicode_escape'))
ValueError: bad marshal data (unknown type code)

I also tried Keras version 1.2.2, same problem.

0 boxes found

@hollance Thanks for your great work. Fantastic writeup. However, I am running into a small issue.

I've successfully trained the NFPA tiny-yolo model with darknet. Great news is I can successfully find an NFPA tag within an image after the model was trained with the darknet detector command. However, after converting to .h5 with yad2k, i run the test with the converted model against the same test image and 0 boxes are found.

Note that I am able to successfully test the yolo (not the tiny yolo) model pre and post convert and it works. What am I missing?

Summary of results below:
a) pre converted tiny-yolo model (success)
b) post converted tiny-yolo model (fail)
c) pre converted yolo model (success)
d) pre converted yolo model (success)

//a) testing pre converted tiny yolo model:
./darknet detector test cfg/NFPA.data cfg/tiny-yolo-NFPA.cfg backup/tiny-yolo-NFPA_6100.weights data/nfpadrum.jpg

layer filters size input output
0 conv 16 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 16
1 max 2 x 2 / 2 416 x 416 x 16 -> 208 x 208 x 16
2 conv 32 3 x 3 / 1 208 x 208 x 16 -> 208 x 208 x 32
3 max 2 x 2 / 2 208 x 208 x 32 -> 104 x 104 x 32
4 conv 64 3 x 3 / 1 104 x 104 x 32 -> 104 x 104 x 64
5 max 2 x 2 / 2 104 x 104 x 64 -> 52 x 52 x 64
6 conv 128 3 x 3 / 1 52 x 52 x 64 -> 52 x 52 x 128
7 max 2 x 2 / 2 52 x 52 x 128 -> 26 x 26 x 128
8 conv 256 3 x 3 / 1 26 x 26 x 128 -> 26 x 26 x 256
9 max 2 x 2 / 2 26 x 26 x 256 -> 13 x 13 x 256
10 conv 512 3 x 3 / 1 13 x 13 x 256 -> 13 x 13 x 512
11 max 2 x 2 / 1 13 x 13 x 512 -> 13 x 13 x 512
12 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024
13 conv 1024 3 x 3 / 1 13 x 13 x1024 -> 13 x 13 x1024
14 conv 30 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 30
15 detection
Loading weights from backup/tiny-yolo-NFPA_6100.weights...
seen 32
Done!
data/nfpadrum.jpg: Predicted in 0.446766 seconds.
NFPA: 94%
Not compiled with OpenCV, saving to predictions.png instead

//b) testing post converted tiny yolo model:
python3 test_yolo.py model_data/tiny-yolo-NFPA.h5 -a model_data/tiny-yolo-NFPA_anchors.txt -c model_data/NFPA_classes.txt

Using TensorFlow backend.
model_data/tiny-yolo-NFPA.h5 model, anchors, and classes loaded.
Found 0 boxes for nfpadrum.jpg

//c) Testing pre conversion yolo (not tiny yolo) model
./darknet detector test cfg/NFPA.data cfg/yolo-NFPA.cfg-WORKS backup/yolo-NFPA_2700.weights data/nfpadrum.jpg

layer filters size input output
0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32
1 max 2 x 2 / 2 416 x 416 x 32 -> 208 x 208 x 32
2 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64
3 max 2 x 2 / 2 208 x 208 x 64 -> 104 x 104 x 64
4 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128
5 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64
6 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128
7 max 2 x 2 / 2 104 x 104 x 128 -> 52 x 52 x 128
8 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256
9 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128
10 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256
11 max 2 x 2 / 2 52 x 52 x 256 -> 26 x 26 x 256
12 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512
13 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256
14 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512
15 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256
16 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512
17 max 2 x 2 / 2 26 x 26 x 512 -> 13 x 13 x 512
18 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024
19 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512
20 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024
21 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512
22 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024
23 conv 1024 3 x 3 / 1 13 x 13 x1024 -> 13 x 13 x1024
24 conv 1024 3 x 3 / 1 13 x 13 x1024 -> 13 x 13 x1024
25 route 16
26 conv 64 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 64
27 reorg / 2 26 x 26 x 64 -> 13 x 13 x 256
28 route 27 24
29 conv 1024 3 x 3 / 1 13 x 13 x1280 -> 13 x 13 x1024
30 conv 30 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 30
31 detection
Loading weights from backup/yolo-NFPA_2700.weights...
seen 32
Done!
data/nfpadrum.jpg: Predicted in 0.573800 seconds.
NFPA: 88%
Not compiled with OpenCV, saving to predictions.png instead

//d) Testing post conversion yolo (not tiny yolo) model
python3 test_yolo.py model_data/yolo-NFPA.h5 -a model_data/yolo-NFPA_anchors.txt -c model_data/NFPA_classes.txt

model_data/yolo-NFPA.h5 model, anchors, and classes loaded.
Found 1 boxes for nfpadrum.jpg
NFPA 0.91 (144, 172) (251, 256)

Thanks!
Todd

link to test image file nfpadrum.jpg
link to cfg file tiny-yolo-NFPA.cfg
link to weights file tiny-yolo-NFPA_6100.weights

Error when converting YOLOv2 model to Metal format.

When I try running the nngraph.py script, I get an error in line 28 saying that the index is out of range. Is there no batch normalization layer here? How do I fix this?

How to add the account for team?

the error come when i buding in my iPhone

"No account for team "MGCEKGP4Y4". Add a new account in the Accounts preference pane or verify that your accounts have valid credentials."

Adding your own model

Thank you for this! How can we take this a step further and add our own model (still YOLO but trained with custom classes - https://github.com/AlexeyAB/darknet)

I have managed to convert the trained model to the .mlmodel format.

Thanks

is there anyway to convert TinyYolo to coreML (not a Tiny-Yolo-voc)

Hi,

I used the converter file for a VOC (tiny-yolo-voc.h5), and every time it gave me an error on my TinyYolo.

is tiny-yolo-voc.h5 a cause this problem?

Can I simple change the dataset to COCO?

I know the original Tiny-Yolo was trained on dataset VOC, however, I want to add more categories by using this net. Can I just simply change the dataset and retrain the model??

Can you add data training your need model? thankyou

Understanding the computation with anchor boxes

Hi, I am not able to understand how the computation for the bounding boxes along with the anchors are happening.
let w = exp(tw) * anchors[2*b ] * blockSize let h = exp(th) * anchors[2*b + 1] * blockSize
Helper.swift

1. How were these anchor boxes selected ?
2. What was the reason to multiply with the exponential function before scaling with the anchor box value?
   The second part is what I am more intrigued by. Any help would be highly appreciated.

Route and Reorg layers with MPSNNGraph

Hi, Thanks for the great writeup and tutorial.
I was trying to implement full V2 version of YOLO with 30 layers .
The previous way, by using MPSTemporaryImages, we would use the offset in source and destinationFeatureOffset parameters for the reorg and route layers(concatenation).
But in the MPSNNGraph, with each layer being a MPSCNNConvolutionNode or MPSCNNPoolingMaxNode , there is no way to specify these params for these nodes.
So would the solution be to add MPSTemporaryImages inbetween these layers whenever needed to create the same effect as the route and reorg layers?

Adding orientation changes - Effect on Bounding boxes

Thank you for your work on this. I am just working on incorporating orientation changes, so that the the camera feed rotates as the screen is rotated while still bounding objects. Im working with the front camera on my iPad pro and everything works perfectly in portrait mode (as it should).
To allow for the landscape orientation I add "Landscape (left of home button)" and "Landscape (right of home button)" in the info.plist.
I also change line 50 previewLayer.connection?.videoOrientation = .portrait to previewLayer.connection?.videoOrientation = .landscapeRight and line 66 videoOutput.connection(with: AVMediaType.video)?.videoOrientation = .portrait to videoOutput.connection(with: AVMediaType.video)?.videoOrientation = .landscapeRight
in https://github.com/hollance/YOLO-CoreML-MPSNNGraph/blob/master/TinyYOLO-CoreML/TinyYOLO-CoreML/VideoCapture.swift

While this gets the screen rotation working, the bounding boxes become a lot bigger than the objects being shown. How can I incorporate screen rotation while keeping the objects still tightly bounded.

Thank you

AttributeError: 'module' object has no attribute 'NetGraph'

Thanks for the amazing tutorial!

I'm trying to convert tiny yolo to a coreml .mlmodel. After I load the model and weights, I try to convert it and get this error. I'm running python2.7, keras 2.1.3, tensorflow 1.5.0, and coremltools 0.7. Any help would be greatly appreciated! Thanks

---

AttributeError Traceback (most recent call last)
in ()
84
85 if name == "main":
---> 86 main()

in main()
78 model.load_weights(weights)
79 print("weights loaded")
---> 80 convert_to_coreml(model)
81 print("converted")
82

in convert_to_coreml(model)
57 image_input_names='image',
58 output_names='grid',
---> 59 image_scale=1/255.)
60

/Users/travis/coreml/lib/python2.7/site-packages/coremltools/converters/keras/_keras_converter.pyc in convert(model, input_names, output_names, image_input_names, is_bgr, red_bias, green_bias, blue_bias, gray_bias, image_scale, class_labels, predicted_feature_name, model_precision, predicted_probabilities_output, add_custom_layers, custom_conversion_functions)
720 predicted_probabilities_output,
721 add_custom_layers,
--> 722 custom_conversion_functions=custom_conversion_functions)
723
724 return _MLModel(spec)

/Users/travis/coreml/lib/python2.7/site-packages/coremltools/converters/keras/_keras_converter.pyc in convertToSpec(model, input_names, output_names, image_input_names, is_bgr, red_bias, green_bias, blue_bias, gray_bias, image_scale, class_labels, predicted_feature_name, model_precision, predicted_probabilities_output, add_custom_layers, custom_conversion_functions)
525 predicted_probabilities_output=predicted_probabilities_output,
526 add_custom_layers=add_custom_layers,
--> 527 custom_conversion_functions=custom_conversion_functions)
528 else:
529 raise RuntimeError(

/Users/travis/coreml/lib/python2.7/site-packages/coremltools/converters/keras/_keras2_converter.pyc in _convert(model, input_names, output_names, image_input_names, is_bgr, red_bias, green_bias, blue_bias, gray_bias, image_scale, class_labels, predicted_feature_name, predicted_probabilities_output, add_custom_layers, custom_conversion_functions)
170
171 from topology2 import NetGraph
--> 172
173 # Build network graph to represent Keras model
174 graph = _topology2.NetGraph(model)

AttributeError: 'module' object has no attribute 'NetGraph'

How do you deal with tf.space_to_depth during coreMLTool conversion?

This is about yolo v2 (not the tiny).

The coreMLTool python part ran without any error generating the .mlmodel file. But when I dropped this into my Xcode project, I got an error:

There was a problem decoding this CoreML document
validator error: Custom layer lambda_1 has an empty 'className' field. This field is required in order for Core ML to link to the implementation for this custom class.

The lambda_1 is the custom layer that calls tf.space_to_depth(x, block_size=2). Do you have any suggestion on what to do?

Why is the feature set 255x13x13 ?

I am trying to fit in a model trained on the VOC dataset with 20 classes. Can you quickly point me to the necessary changes ?

The changes I did so far are:

1. Change the label and anchor in Helpers.swift
2. Change the input and output name for the model built in YOLO.swift
3. Changed numClasses to 20 in YOLO.swift

Not sure about what the gridHeight and gridWidth variables represent. Arent they a single value. My final predictions are 13x13x125.

Cannot run it on my phone

TinyYOLO-CoreML only built successfully using simulator but I got error when I tried it on my iPhone6s. My Xcode version is 10 and my iphone has ios 12. I got two error messages:
1.
Showing Recent Messages
Build system information
error: No account for team "MGCEKGP4Y4". Add a new account in the Accounts preference pane or verify that your accounts have valid credentials. (in target 'TinyYOLO-CoreML')
2.

Showing Recent Messages
No profiles for 'net.machinethink.TinyYOLO-CoreML' were found: Xcode couldn't find any iOS App Development provisioning profiles matching 'net.machinethink.TinyYOLO-CoreML'. (in target 'TinyYOLO-CoreML')

Build failed. How do I fix this? Thank you

Quick Question

It does not seem the ios project has Yolo work for a landscape mode other than portrait. Is this true? (I modified the code to enable landscape rotations in the views and everything, it works fine on portrait).

Quick Question about the model

Is there a way that we can add more objects? If I have pictures to train the model?

Swift Version Error - Expected 3.0,4.0

Error preparing CoreML model "TinyYOLO.mlmodel" for code generation: coremlcompiler: Error: unrecognized Swift version "4.2". Expected one of: 3.0, 3.1, 3.2 or 4.0
coremlcompiler: Error: usage: coremlcompiler [options ...]

Getting Swift Version Error. It says it Expected 3.0 or 4.0 and it is apparently running 4.2

Using the front camera for detection causing boxes to be mirrored

Thank you for the great work here. I am constantly trying to experiment with variations of your work for my own purposes and for one of my applications I am trying to use the object dectection through my front camera (on my iPad) so the I can see what being detected while moving in front my my screen.
I have made the following change in VideoCapture.swift to capture the video stream through the front camera:

Original code chunk:

 guard let captureDevice = AVCaptureDevice.default(for: AVMediaType.video) else {
      print("Error: no video devices available")
      return false
    }

My changed chunk:

    guard let captureDevice = AVCaptureDevice.default(.builtInWideAngleCamera, for: AVMediaType.video, position: .front) else {
        print ("Error: no video devices available")
        return false
    }

All works well and good except that the bounding boxes are mirrored, meaning that if I am holding an apple in my left hand and a banana in my right hand, the boxes are swapped around these objects. I understand that this might have to do something with the iPhones front camera mirroring the video display but I am not sure how to tackle this issue. Any guidance will be very useful. Thank you

A Question

Hi Matt,

I’ve read your blog about this topic and found it very informative. It did leave me wondering if there’s a way to use an .MLModel file without vison. I look forward to hearing back from you

is it possible to use multiple bounding box using Inceptionv3 model?

Yolo has grid method but InceptionV3 does not.
MLMultiArray
`
class TinyYOLOOutput : MLFeatureProvider {
let grid: MLMultiArray

var featureNames: Set<String> {
    get {
        return ["grid"]
    }
}

func featureValue(for featureName: String) -> MLFeatureValue? {
    if (featureName == "grid") {
        return MLFeatureValue(multiArray: grid)
    }
    return nil
}

init(grid: MLMultiArray) {
    self.grid = grid
}

}
`

Showing only one bounding box per frame (coreml)

I am trying to find a way to only show one bounding box per frame, meaning that I dont want it to detect three "persons" in one frame. In the view controller I have modified the maxInflightBuffers to 1 but I am not sure if that is the way to go. The reason of such implementation, I am enabling an audio file when a certain bounding box appears, but if there is more than one, it wont play.

Boxes remain on screen after moving on from objects

Thank you so much for your work here. I have began to notice the issue of bounding boxes persisting on my iphone 6s screen even after I point to another object. I am using my own ML model that I converted from custom training with darknet, I have using this model for a few days now and today is the first time this issue has come up. I reinstalled the app with my coreML graph on with no real luck yet. Any pointers as to why this might be?
The app is running at about 11 FPS so its clearly not an obvious lag. Any guidance will be very helpful. Thanks

The conversion doesn't seem to work for me

Hi!

I have a darknet model that is based on tiny-yolo-voc and finds license plates pretty accurately (checked that many times with darknet's detector command).

Your conversion mechanism works pretty well for your model (just like you write in readme).

However, when I apply your conversion process to my darknet model, the resulting coreml model doesn't seem to be able to recognise anything.

Would you be kind enough to help me in any way?

Failed to load Keras model .h5

Hi I always getting the issue when trying to convert Keras to CoreML. I tried multiple solutions like switching among Keras 1.2.2 , 2.0.6 and 2.0.8. The Keras model is my own trained model using darknet and yolo-voc.2.0.cfg and It was converted to Keras successfully as the first step of your tutorial.

File "coreml.py", line 11, in
image_scale=1/255.)
File "/Users/ln160c/Downloads/YOLO-CoreML-MPSNNGraph-master/Convert/coreml/lib/python2.7/site-packages/coremltools/converters/keras/_keras_converter.py", line 489, in convert
predicted_probabilities_output = predicted_probabilities_output)
File "/Users/ln160c/Downloads/YOLO-CoreML-MPSNNGraph-master/Convert/coreml/lib/python2.7/site-packages/coremltools/converters/keras/_keras_converter.py", line 158, in _convert
model = _keras.models.load_model(model)
File "/Users/ln160c/Downloads/YOLO-CoreML-MPSNNGraph-master/Convert/coreml/lib/python2.7/site-packages/keras/models.py", line 142, in load_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/Users/ln160c/Downloads/YOLO-CoreML-MPSNNGraph-master/Convert/coreml/lib/python2.7/site-packages/keras/models.py", line 193, in model_from_config
return layer_from_config(config, custom_objects=custom_objects)
File "/Users/ln160c/Downloads/YOLO-CoreML-MPSNNGraph-master/Convert/coreml/lib/python2.7/site-packages/keras/utils/layer_utils.py", line 40, in layer_from_config
custom_objects=custom_objects)
File "/Users/ln160c/Downloads/YOLO-CoreML-MPSNNGraph-master/Convert/coreml/lib/python2.7/site-packages/keras/engine/topology.py", line 2582, in from_config
process_layer(layer_data)
File "/Users/ln160c/Downloads/YOLO-CoreML-MPSNNGraph-master/Convert/coreml/lib/python2.7/site-packages/keras/engine/topology.py", line 2560, in process_layer
custom_objects=custom_objects)
File "/Users/ln160c/Downloads/YOLO-CoreML-MPSNNGraph-master/Convert/coreml/lib/python2.7/site-packages/keras/utils/layer_utils.py", line 40, in layer_from_config
custom_objects=custom_objects)
File "/Users/ln160c/Downloads/YOLO-CoreML-MPSNNGraph-master/Convert/coreml/lib/python2.7/site-packages/keras/layers/core.py", line 682, in from_config
function = func_load(config['function'], globs=globs)
File "/Users/ln160c/Downloads/YOLO-CoreML-MPSNNGraph-master/Convert/coreml/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 183, in func_load
code = marshal.loads(code.encode('raw_unicode_escape'))
ValueError: bad marshal data (unknown type code)

FPS on ipad or ipad pro

Hi,
I'm not sure if this is the correct place for questions to be asked, apologies if not. I am designing an app that would require Yolo to run at, or close to the same frame rate as video. Would this solution get anywhere close to 30fps on Ipad or Ipad pro?

Thanks

Compatibility with YOLOV2

Hi there, Thank you very much for you work on this. I am attempting to use my YOLOv2 model on the app in pursuit of a higher accuracy. I am able to successfully convert the YOLO weights into a mlCore model, but when I add the model onto the xcode, there is no recognition happening. I know the model is loaded because the FPS dropped from ~19 while using tiny-yolo to ~3 on my iPhone 8. Is there any other change that needs to be made when working with a yolo model? The model size is about 280mb and I have already made changes to reflect in the number of classes and their labels.
Thank you.

Keras layer '<class 'keras.layers.core.Lambda'>' not supported

This is the full error:

Traceback (most recent call last):
  File "coreml-stopsign.py", line 10, in <module>
    output_names='grid')
  File "/home/sean/.local/lib/python3.6/site-packages/coremltools/converters/keras/_keras_converter.py", line 505, in convert
    predicted_probabilities_output = predicted_probabilities_output)
  File "/home/sean/.local/lib/python3.6/site-packages/coremltools/converters/keras/_keras2_converter.py", line 166, in _convert
    _check_unsupported_layers(model)
  File "/home/sean/.local/lib/python3.6/site-packages/coremltools/converters/keras/_keras2_converter.py", line 89, in _check_unsupported_layers
    "Keras layer '%s' not supported. " % str(type(layer)))
ValueError: Keras layer '<class 'keras.layers.core.Lambda'>' not supported.

It appears that the YAD2K conversion between Darknet and Keras adds a Lambda layer, which is unsupported in Apple's coremltools. How did you work around this?

The lambda layer seems to be added here (yad2k.py):

elif section.startswith('reorg'):
            block_size = int(cfg_parser[section]['stride'])
            assert block_size == 2, 'Only reorg with stride 2 supported.'
            all_layers.append(
                Lambda(
                    space_to_depth_x2,
                    output_shape=space_to_depth_x2_output_shape,
                    name='space_to_depth_x2')(prev_layer))
            prev_layer = all_layers[-1]

Generated coreml file crash on loading

Hi, first of all thank you for providing such an amazing project.
I've tried your demo it works great.
When I tried to generate a new model following the steps and with the original .cfg and .weights file, everything works great, however when I try to load the generated yolo class from YOLO.swift, the app crashed everytime, could you help me if there's anything that I messed up ?
I'm following the steps in readme.
Great thanks.

here is the screenshot:

Question of implementation

Hi, I saw you implementation about Tiny YOLO in

YOLO-CoreML-MPSNNGraph/TinyYOLO-CoreML/TinyYOLO-CoreML/Helpers.swift

I modified some parameters without changing any main logic. However, I found that the confidence result on iPhone is significantly different with computer version.

I got 0.91 in computer version and 0.18 for iPhone version. Do you know what happen?
Computer version: https://github.com/thtrieu/darkflow

Unable to convert the model from keras to CoreML

2019-04-01 19:13:45.971729: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
/Users/ratzzz/PycharmProjects/CoreML/env/lib/python2.7/site-packages/keras/models.py:245: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.
warnings.warn('No training configuration found in save file: '
Traceback (most recent call last):
File "coreml.py", line 8, in
image_scale=1/255.)
File "/Users/ratzzz/PycharmProjects/CoreML/env/lib/python2.7/site-packages/coremltools/converters/keras/_keras_converter.py", line 760, in convert
custom_conversion_functions=custom_conversion_functions)
File "/Users/ratzzz/PycharmProjects/CoreML/env/lib/python2.7/site-packages/coremltools/converters/keras/_keras_converter.py", line 556, in convertToSpec
custom_objects=custom_objects)
File "/Users/ratzzz/PycharmProjects/CoreML/env/lib/python2.7/site-packages/coremltools/converters/keras/_keras2_converter.py", line 209, in _convert
graph.build()
File "/Users/ratzzz/PycharmProjects/CoreML/env/lib/python2.7/site-packages/coremltools/converters/keras/_topology2.py", line 750, in build
self.defuse_activation()
File "/Users/ratzzz/PycharmProjects/CoreML/env/lib/python2.7/site-packages/coremltools/converters/keras/_topology2.py", line 510, in defuse_activation
isinstance(k_layer, _keras.layers.SeparableConv1D) or
AttributeError: 'module' object has no attribute 'SeparableConv1D'

I am using Keras 2.0.6

Could you please help on this issue?