tucan9389/objectdetection-coreml

An example running Object Detection using Core ML (YOLOv8, YOLOv5, YOLOv3, MobileNetV2+SSDLite)

ObjectDetection-CoreML

supporting models: YOLOv8, YOLOv5, YOLOv3, MobileNetV2+SSDLite

This project is Object Detection on iOS with Core ML.
If you are interested in iOS + Machine Learning, visit here you can see various DEMOs.

Requirements

• Xcode 10.3+
• iOS 13.0+
• Swift 4.2

How To Build and Run the Project

1. Clone the project

git clone https://github.com/tucan9389/ObjectDetection-CoreML

2. Prepare Core ML model

• You can download COCO models or another model from here

Or if you want to make and use model with custom dataset,

1. follow roboflow tutorial from scratch or yolov5 repo's tutorial
2. and convert the .pt model to .mlmodel model with our issue.

3. Add the model to the project

By default, the project uses the yolov8s model. If you want to use another model, you can replace the model file in the project.

4. Set model name properly in ViewController.swift

5. Build and Run

How To Run with your own model

1. Convert your model to Core ML

At this moment(23.04.08), there is error when converting yolov8 models to Core ML. Once ultralytics/ultralytics#1791 is merged, you can use the following steps. (Or you can use this PR alternatively.)

Pre-requirements

pip install ultralytics
pip install coremltools

Option 1) With shell

yolo export model=yolov8n.pt format=coreml nms

Option 2) With python script

# mian.py
from ultralytics import YOLO

if __name__ == '__main__':
    model = YOLO("yolov8n.pt", task='detect')  # load a pretrained model
    model.overrides['nms'] = True
    success = model.export(format="coreml")  # export the model to CoreML format

# in terminal
python main.py
# then you can see the `.mlpackage` or `.mlmodel` file in your current directory
# (btw you can check your current directory with `pwd` command)

2. Follow the steps above from Step 3

Models

Model Matadata

Model Size, Minimum iOS Version, Download Link

Model	Size (MB)	Minimum iOS Version	Download Link	Trained Dataset
yolov8n.mlmodel	12.7	iOS14	Link
yolov8s.mlmodel	44.7	iOS14	Link
yolov8m.mlmodel	52.1	iOS14	Link
yolov8l.mlmodel	87.8	iOS14	Link
yolov8x.mlmodel	272.9	iOS14	Link
yolov5n.mlmodel	7.52	iOS13	Link	COCO
yolov5s.mlmodel	28.0	iOS13	Link	COCO
yolov5m.mlmodel	81.2	iOS13	Link	COCO
yolov5l.mlmodel	178.0	iOS13	Link	COCO
yolov5x.mlmodel	331.0	iOS13	Link	COCO
yolov5n6.mlmodel	12.8	iOS13	Link	COCO
yolov5s6.mlmodel	48.5	iOS13	Link	COCO
yolov5m6.mlmodel	137.0	iOS13	Link	COCO
yolov5l6.mlmodel	293.0	iOS13	Link	COCO
yolov5x6.mlmodel	537.0	iOS13	Link	COCO
YOLOv3.mlmodel	248.4	iOS12	Link	COCO
YOLOv3FP16.mlmodel	124.2	iOS12	Link	COCO
YOLOv3Int8LUT.mlmodel	62.2	iOS12	Link	COCO
YOLOv3Tiny.mlmodel	35.5	iOS12	Link	COCO
YOLOv3TinyFP16.mlmodel	17.8	iOS12	Link	COCO
YOLOv3TinyInt8LUT.mlmodel	8.9	iOS12	Link	COCO
MobileNetV2_SSDLite.mlmodel	9.3	iOS12	Link	COCO
ObjectDetector.mlmodel	63.7	iOS12	Link	6 Label Dataset

Trained Dataset Infos

COCO Dataset

• https://github.com/ultralytics/yolov5/blob/9da6d0f9f5bc37fa386b7b82d2a963f94650949a/data/coco.yaml#L17-L97

6 Label Dataset(Apple's DEMO)

• Bagel
• Banana
• Coffee
• Croissant
• Egg
• Waffle

Performance

Build Setting:
Xcoede > Build Settings > Apple Clang - Code Generation > Optimization Level > Fastest [-O3]

Infernece Time (ms)

Model vs. Device	14 Pro	13 Pro	12 Pro	11 Pro	XS	XS Max	XR	X	7+	7
yolov8n	15
yolov8s	29
yolov8m	37
yolov8l	45
yolov8x	51
yolov5n			24
yolov5s			29
yolov5m			39
yolov5l			38
yolov5x			69
yolov5n6			24
yolov5s6			34
yolov5m6			39
yolov5l6			41
yolov5x6			57
YOLOv3			45	83	108	93	100	356	569	561
YOLOv3FP16			44	84	104	89	101	348	572	565
YOLOv3Int8LUT			53	86	101	92	100	337	575	572
YOLOv3Tiny			36	44	46	41	47	106	165	168
YOLOv3TinyFP16			33	44	51	41	44	103	165	167
YOLOv3TinyInt8LUT			39	44	45	39	39	106	160	161
MobileNetV2_SSDLite			17	18	31	31	31	109	141	134
ObjectDetector			13	18	24	26	23	63	86	84

Total Time (ms)

Model vs. Device	14 Pro	13 Pro	12 Pro		11 Pro	XS	XS Max	XR	X	7+	7
yolov8n	15
yolov8s	31
yolov8m	39
yolov8l	47
yolov8x	52
yolov5n			26
yolov5s			31
yolov5m			41
yolov5l			39
yolov5x			72
yolov5n6			25
yolov5s6			36
yolov5m6			41
yolov5l6			42
yolov5x6			59
YOLOv3			46	84	108	93	100	357	569	561
YOLOv3FP16			45	85	104	89	101	348	572	565
YOLOv3Int8LUT			54	86	102	92	102	338	576	573
YOLOv3Tiny			37	45	46	42	48	106	166	169
YOLOv3TinyFP16			35	45	51	41	44	104	165	167
YOLOv3TinyInt8LUT			41	45	45	39	40	107	160	161
MobileNetV2_SSDLite			19	19	32	31	32	109	142	134
ObjectDetector			14	18	25	26	23	64	87	85

FPS

Model vs. Device	14 Pro	13 Pro	12 Pro		11 Pro	XS	XS Max	XR	X	7+	7
yolov8n	38
yolov8s	14
yolov8m	14
yolov8l	14
yolov8x	13
yolov5n			19
yolov5s			14
yolov5m			13
yolov5l			14
yolov5x			7
yolov5n6			19
yolov5s6			14
yolov5m6			13
yolov5l6			14
yolov5x6			13
YOLOv3			12	9	8	10	9	2	1	1
YOLOv3FP16			13	9	9	10	8	2	1	1
YOLOv3Int8LUT			14	9	9	10	9	2	1	1
YOLOv3Tiny			14	14	21	22	20	8	5	5
YOLOv3TinyFP16			14	14	19	23	21	9	5	5
YOLOv3TinyInt8LUT			11	14	21	24	23	8	5	5
MobileNetV2_SSDLite			19	29	23	23	23	8	6	6
ObjectDetector			17	29	23	23	24	14	10	11

See also

• motlabs/awesome-ml-demos-with-ios
  : The challenge using machine learning model created from tensorflow on iOS
• Machine Learning - Models - Apple Developer
• hollance/coreml-survival-guide
• vonholst/SSDMobileNet_CoreML
  : iOS project for object detection(SSDMobileNet V1) using Core ML.
• ultralytics/ultralytics
  : YOLOv8 repository
• ultralytics/yolov5
  : YOLOv5 repository

objectdetection-coreml's Issues

images for README

Support YOLOv8

yolov8: https://github.com/ultralytics/ultralytics

At this moment(23.04.08), there is error when converting yolov8 models to Core ML. Once ultralytics/ultralytics#1791 is merged, you can use the following steps. (Or you can use this PR alternatively.)

Pre-requirements

pip install ultralytics
pip install coremltools

Option 1) With shell

yolo export model=yolov8n.pt format=coreml nms

Option 2) With python script

# mian.py
from ultralytics import YOLO

if __name__ == '__main__':
    model = YOLO("yolov8n.pt", task='detect')  # load a pretrained model
    model.overrides['nms'] = True
    success = model.export(format="coreml")  # export the model to CoreML format

# in terminal
python main.py
# then you can see the `.mlpackage` or `.mlmodel` file in your current directory
# (btw you can check your current directory with `pwd` command)

Are there any parameters in iOS that need to be adjusted to optimize real-time detection

Hi @tucan9389 :

Thanks for your work firstly!

I have successfully converted the yolov5s model into an .mlmodel and deployed it using your code.
My model has a good effect on static images, but no result or inaccurate position is found in real-time detection. May I ask what is the possible reason? Do I need to make some modifications in the iOS app code?
The interested thing is, my model also performed well with video testing locally, so I was confused about what went wrong.

Thanks for your help!

Exporting from yolov5 to coreml

having trouble exporting our ultralytics yolov5 models to coreml.

git clone repo, cd repo, checkout fix/coreml_export_nms_layer, install req, run export cmd

python3 export.py --weights ../../output/best.pt --include coreml

... lots of stuff happening, and eventually ...

added const 10 for builder <coremltools.models.neural_network.builder.NeuralNetworkBuilder object at 0x7f078bb94580>
added const 11 for builder <coremltools.models.neural_network.builder.NeuralNetworkBuilder object at 0x7f078bb94580>
added const 12 for builder <coremltools.models.neural_network.builder.NeuralNetworkBuilder object at 0x7f078bb94580>
Translating MIL ==> NeuralNetwork Ops: 97%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 656/678 [00:00<00:00, 1012.99 ops/s]added const 878 for builder <coremltools.models.neural_network.builder.NeuralNetworkBuilder object at 0x7f078bb94580>
Translating MIL ==> NeuralNetwork Ops: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 678/678 [00:00<00:00, 1160.80 ops/s]

CoreML: export failure: 0 has type int, but expected one of: bytes, unicode
/usr/src/app/temp/yolov5#

if its helpful we can share our best.pt.

build iOS app error

there is a bug report when build the project:
libc++abi: terminating with uncaught exception of type NSException
*** Terminating app due to uncaught exception 'NSGenericException', reason: '*** -[AVCaptureSession startRunning] startRunning may not be called between calls to beginConfiguration and commitConfiguration'
terminating with uncaught exception of type NSException
CoreSimulator 857.14 - Device: iPhone 14 (CA4849C9-1866-4D53-BCEA-B0ED86A38F30) - Runtime: iOS 16.2 (20C52) - DeviceType: iPhone 14

if you have time , pls fix it. I am new to swift , thank you.

Yolo8 to coreml conversion

from ultralytics import YOLO

model=YOLO('runs/detect/train/weights/best.pt')

model.export(format='coreml',nms=True)

CoreML: export failure ❌ 9.9s: 1 name found for nc=8395 CoreML model is not converted.

I have found this error when I use nms=True, I need the model with bounding box.
Could you help me ?

Yolo v4 or v5

Do you know if this works with Yolo v4 or V5?

Inference time, execution time, fps

Infernece Time (ms)

Model vs. Device	11 Pro	XS	XS Max	XR	X	7+	7
YOLOv3	83	108	93	100	356	569	561
YOLOv3FP16	84	104	89	101	348	572	565
YOLOv3Int8LUT	86	101	92	100	337	575	572
YOLOv3Tiny	44	46	41	47	106	165	168
YOLOv3TinyFP16	44	51	41	44	103	165	167
YOLOv3TinyInt8LUT	44	45	39	39	106	160	161
MobileNetV2_SSDLite	18	31	31	31	109	141	134
ObjectDetector	18	24	26	23	63	86	84

Total Time (ms)

Model vs. Device	11 Pro	XS	XS Max	XR	X	7+	7
YOLOv3	84	108	93	100	357	569	561
YOLOv3FP16	85	104	89	101	348	572	565
YOLOv3Int8LUT	86	102	92	102	338	576	573
YOLOv3Tiny	45	46	42	48	106	166	169
YOLOv3TinyFP16	45	51	41	44	104	165	167
YOLOv3TinyInt8LUT	45	45	39	40	107	160	161
MobileNetV2_SSDLite	19	32	31	32	109	142	134
ObjectDetector	18	25	26	23	64	87	85

FPS

Model vs. Device	11 Pro	XS	XS Max	XR	X	7+	7
YOLOv3	9	8	10	9	2	1	1
YOLOv3FP16	9	9	10	8	2	1	1
YOLOv3Int8LUT	9	9	10	9	2	1	1
YOLOv3Tiny	14	21	22	20	8	5	5
YOLOv3TinyFP16	14	19	23	21	9	5	5
YOLOv3TinyInt8LUT	14	21	24	23	8	5	5
MobileNetV2_SSDLite	29	23	23	23	8	6	6
ObjectDetector	29	23	23	24	14	10	11

Problem with detection on IPhone 15 Pro

Hello,
I deploy the application on an IPhone 11 and an IPhone 13 Pro and all worked well.
But when I deploy on an IPhone 15 Pro and nothing work. Sometimes the app find something but it's false.
Do you have any idea on what happens ?
Thanks.

Why IPAD with M1 is pretty much slower than Iphone 13 Pro with A15??

I run your models like YoloV8x ~ s and DeeplabV3, FRCN on both iphone and iPad.
But always it shows, Ipad is about 2~3 times slower than iphone.
Could you give me some hint the reason or any guess??

Thank you

Error when trying to run the code

After downloading the code I tried to run it, however the output is an error message: "cannot find 'yolov5s' in scope". Can you help me with that?

Support various TF object detection model

https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md

Support YOLOv5

Saving Predicted Video

I am able to run the yolo model in Live streaming . Can you just help me out in saving the frames to video in LIVE CAPTURE() after prediction

How can I use my own created .mlmodel instead of yolov3?

Hi, how can I switch the .mlmodel file so I can use my own one created with the MLImageClassifierBuilder?

I tried to switch the .mlmodel files and renamed the files so it could get resolved and the app runs but the app then can not detect anything. Do I have to consider anything else when I want to use my own .mlmodel file?

Convert method

Could you share about the method that convert tf-frozen-model to coreml-model?