infocom-tpo/swiftopenpose

tf-openpose Based iOS Project

SwiftOpenPose

This project was developed by transplanting tf-openpose to Swift.
Community cooperation is welcome.

Environment

• iOS11
• Xcode9

Training Model

There are two learning models available for this project.

• OpenPose Caffe-Model
  • Convert from OpenPose Caffe-Model to MLModel
• tf-openpose Mobilenet Model (instance_normalization Disabled Version)
  • infocom-tpo/tf-openpose .. Model Training and Converter
  • MobileOpenPose.mlmodel .. Model Download

Performance comparison

• BenchMark Hardware: iPad 2017
  • OpenPose Caffe-Model
    • processing time .. range 2-4 Sec.
  • tf-openpose Mobilenet Model
    • processing time .. Less than 1 sec

Dependencies Library

• UpSurge
• OpenCV
  • Download of iOS Pack
  • Opencv lightweight version

Installation

$ git clone https://github.com/infocom-tpo/SwiftOpenPose.git
$ cd SwiftOpenPose
$ pod install
$ curl -o SwiftOpenPose/Resources/MobileOpenPose.mlmodel \
https://s3-ap-northeast-1.amazonaws.com/swiftopenpose/MobileOpenPose.mlmodel

Examples

• Bone Detecter
  This app exports a video with detected bones to photo library.
  You can shoot or select a video to detect bones.

iTunes Store App

Stickman Animator

Stickman Animator is an app to make animations of stickman from videos of people.

Blogs

• Explanation by Japanese
  • AI初心者がOpenPoseのモバイル実装に取り組んだ話

Reference

• OpenPose
• tf-openpose
• OpenPose Caffe Model Convert to CoreML Model
• エネルギー波を繰り出す女子高生

Development By Infocom TPO

Infocom TPO

License

SwiftOpenPose is available under the MIT license. See the LICENSE file for more info.

Citation

@inproceedings{cao2017realtime,
  author = {Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh},
  booktitle = {CVPR},
  title = {Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},
  year = {2017}
}

swiftopenpose's Issues

How can i run this source code with XCode?

A code question about the Matrix operation in Common.swift

Hello, I am confused about the part in Common.swift that
heatMat = Matrix<Double>( (0..<heatMat.rows).map({ heatMat.row($0) - min(heatMat.row($0)) }))
My understanding is that among the range of 0..<heatMat.rows(19), let every element in each heatMat.row minus the minimum element of it's own row.

For instance, row1 = [6.0,4.0,3.0,1.0], after this operation got row1 = [5.0,3.0,2.0,1.0]?

Could you please help me understand what is the real operation of "heatMat.row($0)-min(heatMat.row($0))"? and What's the reason to obtain this new heatMat?

Thank you very much!

What Parts of the Body Correspond to the Model Output

First of all, great work with this implementation! Thanks for sharing the code!

I wrote code using the original openpose implementation and am trying to use the mobile version to run it on an ios device.

I am getting confused with the model output (shown below):

It seems that the mobile version of openpose outputs 10 segments per frame (with each segment having a start and end coordinate).

However the full version of openpose outputs 25 coordinates for each frame according to the following format.

Is there a similar diagram for mobile openpose which maps the keypoints generated by the model to their location on the human body?

Thanks!

Is there any way to get a toe keypoint?

Hello

I've noticed that the OpenPose model found here includes the body-foot estimation:
https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation.md#installation

I'm not sure which model you have based this project on, but is there any way to include the foot keypoint? Does it already exist in your model?

@mbotsu - any insight on this? Any information would be greatly appreciated!

How can i run this project with XCode?

Screenshots:
http://prntscr.com/hxh7vg
http://prntscr.com/hxh7zc

XCode 9.2

How to generate the mobilenet.mlmodel file?

Thank you for this awesome project!
When I compile the project, I got this error:
coremlc: Error: unable to read document: /Users/qyvideo/work/deep_learning/SwiftOpenPose/SwiftOpenPose/Resources/mobilenet.mlmodel
detail: validator error: unable to open file for read
and how to generate this file? Is it in this section:
OpenPose Keras Mobilenet-Model
Result: However it did not work.

how to support the hand model

Hello,
is it possible to run the hand model on this project, if not please can you tell what s needed in order to run it
Best
Lafi

How to download the original caffe keras model which MobileOpenPose.mlmodel converted from?

Thanks for your great job first.
I have success run your swift demo on pose detect.
Can you tell me how to download the original caffe keras model which MobileOpenPose.mlmodel converted from?

Thanks.

The scale of coordinates in SwiftOpenPose

How to modify xy-axis coordinates in OpenPose to match the scale in SwiftOpenPose?
I tried to use coco model to get coordinates data ,but I get the different result by the same coco model between OpenPose and SwiftOpenPose.

BoneDetecter did compile and run, but the generated output same as the input

First, your sharing and effort are great!

I am not sure where goes wrong, leading my generated app following your shared code "BoneDetecter", and run on iPhone 8 Plus, does not generate a video that having points and vectors of the Pose, the output video simple the same video I capture or selected from iPhone's album.

Pre-condition:
#1 iPhone 8 Plus with Xcode 10;
#2 Download pod install, and download MobileOpenPose.mlmodel
#3 Run on Phone, the UI allow select or capture video for processing
#4 The Processing taking longer time than your "Stickman Animator" as apple-to-apple compare
#5 The App said video generated, and i go and check the video, it is simply the same as input

Questions:
Q1: Does the shared source code very different to your "Stickman Animator" on AppStore?
Q2: Where I miss, leading the generated video does not contains the recognised vectors?

Many thanks for your sharing again.

An issue for compile

Hi! After I compiled SwiftOpenPose.xcworkspace succeed on Xcode for my iPhone 6s, while I opening the app, it seems like stuck on the image(swiftopenpose_result.png) that on the /SwiftOpenPose/images. Do you know what happened?
Thanks

BoneDetecter did not work. Compile error.

Thank you for this project.
When I compile the example BoneDetecter, I got this error:

./examples/BoneDetecter/Pods/Upsurge/Sources/Upsurge/ValueArray.swift:134:35: Missing argument for parameter 'capacity' in call
./examples/BoneDetecter/Pods/Swift.UnsafeMutablePointer:273:17: 'deallocate(capacity:)' declared here

How can I run BoneDetecter ?

Infocomさんは、確か日本だったかと思いますので、日本語でも記入させていただきます。
すみません、exampleにあるBoneDetecterを動かそうと思い、コンパイルすると、上記のエラーとなります。
何が原因だかお分かりになるでしょうか？
xcode9.2 です。Upsurge、IteratorTools、OpenCVは、ルートフォルダにあったPodファイルを改変作成してインストールしています。

  pod 'Upsurge', '~> 0.10.0'
  pod 'IteratorTools', '~> 1.1.0'
  pod 'OpenCV', '~> 3.3.1'

よろしくお願いします。

What is the bin file in 'Resources' directory?

I want to use this to detect body in my custom picture. but after I drag it into the Resources directory and change a little bit here:
let fname = "test.jpg"
It seems the app didn't respond and can not detect where the body is.
I noticed there are two bin files(hadou.bin and person1.bin). Should I generate a bin file first for my custom picture?

Question : FPS rate

@mganeko How much time it would take to process live video frames @ 60 FPS ?

How to integrate camera with this project