layout | title | nav_order |
---|---|---|
default |
Home |
1 |
This is a fork of MediaPipe to demonstrate building MediaPipe as a Framework for iOS, in this case, the FaceMesh model
- This builds an XCFramework into ./frameworkbuild/FaceMeshIOSLibFramework/xcframework
- The XCFramework contains both arm64 and x86_64 (iOS Simulator) parts, so you can use this on both real devices and on the iOS simulator
- I've created the Objective-C file for the framework in //mediapipe/examples/ios/facemeshioslib
- prerequisites
- You need to have Google's Bazel installed. Personally I install via node (
npm install -g bazel
).
- You need to have Google's Bazel installed. Personally I install via node (
- run
./BUILD_FACE_MESH_XCFRAMEWORK.sh
, the resulting framework should then appear in ./frameworkbuild/FaceMeshIOSLibFramework/xcframework/FaceMeshIOSLibFramework.xcframework- Copy the framework and use it in your projects. You're welcome.
- Framework usage :
#import <FaceMeshIOSLibFramework/FaceMeshIOSLibFramework.h>
- only one class : FaceMeshIOSLib
- delegate callback gives you an array of detected faces (But there's only one face configured in my graph.. so there's at most length 1)
- each face is an array of 468
FaceMeshIOSLibFaceLandmarkPoint
points (x,y,z)
- each face is an array of 468
- delegate callback gives you an array of detected faces (But there's only one face configured in my graph.. so there's at most length 1)
- only one class : FaceMeshIOSLib
- I couldn't have done this without these sources
- This great example : https://powderapp.medium.com/mediapipe-tutorial-find-memes-that-match-your-facial-expression-9bf598da98c0
- Mentioned from issue here
- Mediapipe graph visualizer : https://viz.mediapipe.dev
- Understanding mediapipe graphs/calculators : I find it great to start here https://google.github.io/mediapipe/framework_concepts/calculators.html
- This Stackoverflow thread for pointing out how to create XCFrameworks
- I tried using good ol'
lipo -create ... -output ...
with the binaries and putting it in a normal.framework
. XCode 12.4 refuses to build with the errorBuilding for iOS Simulator, but the linked and embedded framework was built for iOS + iOS Simulator
. It seemed not so long ago that I last used thelipo
method. Things change so fast.
- I tried using good ol'
- This great example : https://powderapp.medium.com/mediapipe-tutorial-find-memes-that-match-your-facial-expression-9bf598da98c0
MediaPipe offers cross-platform, customizable ML solutions for live and streaming media.
Face Detection | Face Mesh | Iris | Hands | Pose | Holistic |
---|---|---|---|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
Hair Segmentation | Object Detection | Box Tracking | Instant Motion Tracking | Objectron | KNIFT |
---|---|---|---|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
Android | iOS | C++ | Python | JS | Coral | |
---|---|---|---|---|---|---|
Face Detection | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
Face Mesh | ✅ | ✅ | ✅ | ✅ | ✅ | |
Iris | ✅ | ✅ | ✅ | |||
Hands | ✅ | ✅ | ✅ | ✅ | ✅ | |
Pose | ✅ | ✅ | ✅ | ✅ | ✅ | |
Holistic | ✅ | ✅ | ✅ | ✅ | ✅ | |
Selfie Segmentation | ✅ | ✅ | ✅ | ✅ | ✅ | |
Hair Segmentation | ✅ | ✅ | ||||
Object Detection | ✅ | ✅ | ✅ | ✅ | ||
Box Tracking | ✅ | ✅ | ✅ | |||
Instant Motion Tracking | ✅ | |||||
Objectron | ✅ | ✅ | ✅ | ✅ | ||
KNIFT | ✅ | |||||
AutoFlip | ✅ | |||||
MediaSequence | ✅ | |||||
YouTube 8M | ✅ |
See also MediaPipe Models and Model Cards for ML models released in MediaPipe.
To start using MediaPipe solutions with only a few lines code, see example code and demos in MediaPipe in Python and MediaPipe in JavaScript.
To use MediaPipe in C++, Android and iOS, which allow further customization of the solutions as well as building your own, learn how to install MediaPipe and start building example applications in C++, Android and iOS.
The source code is hosted in the MediaPipe Github repository, and you can run code search using Google Open Source Code Search.
- Bringing artworks to life with AR in Google Developers Blog
- Prosthesis control via Mirru App using MediaPipe hand tracking in Google Developers Blog
- SignAll SDK: Sign language interface using MediaPipe is now available for developers in Google Developers Blog
- MediaPipe Holistic - Simultaneous Face, Hand and Pose Prediction, on Device in Google AI Blog
- Background Features in Google Meet, Powered by Web ML in Google AI Blog
- MediaPipe 3D Face Transform in Google Developers Blog
- Instant Motion Tracking With MediaPipe in Google Developers Blog
- BlazePose - On-device Real-time Body Pose Tracking in Google AI Blog
- MediaPipe Iris: Real-time Eye Tracking and Depth Estimation in Google AI Blog
- MediaPipe KNIFT: Template-based feature matching in Google Developers Blog
- Alfred Camera: Smart camera features using MediaPipe in Google Developers Blog
- Real-Time 3D Object Detection on Mobile Devices with MediaPipe in Google AI Blog
- AutoFlip: An Open Source Framework for Intelligent Video Reframing in Google AI Blog
- MediaPipe on the Web in Google Developers Blog
- Object Detection and Tracking using MediaPipe in Google Developers Blog
- On-Device, Real-Time Hand Tracking with MediaPipe in Google AI Blog
- MediaPipe: A Framework for Building Perception Pipelines
- MediaPipe Seattle Meetup, Google Building Waterside, 13 Feb 2020
- AI Nextcon 2020, 12-16 Feb 2020, Seattle
- MediaPipe Madrid Meetup, 16 Dec 2019
- MediaPipe London Meetup, Google 123 Building, 12 Dec 2019
- ML Conference, Berlin, 11 Dec 2019
- MediaPipe Berlin Meetup, Google Berlin, 11 Dec 2019
- The 3rd Workshop on YouTube-8M Large Scale Video Understanding Workshop, Seoul, Korea ICCV 2019
- AI DevWorld 2019, 10 Oct 2019, San Jose, CA
- Google Industry Workshop at ICIP 2019, 24 Sept 2019, Taipei, Taiwan (presentation)
- Open sourced at CVPR 2019, 17~20 June, Long Beach, CA
- Awesome MediaPipe - A curated list of awesome MediaPipe related frameworks, libraries and software
- Slack community for MediaPipe users
- Discuss - General community discussion around MediaPipe
MediaPipe is currently in alpha at v0.7. We may be still making breaking API changes and expect to get to stable APIs by v1.0.
We welcome contributions. Please follow these guidelines.
We use GitHub issues for tracking requests and bugs. Please post questions to
the MediaPipe Stack Overflow with a mediapipe
tag.