Comments (6)
There you go. It's probably because the model expects images with faces up. I tried doing frame.rotate(90, 0, 0)
in the frame processor before passing the frame to the face detector but it doesn't seem to have any effect.
RPReplay_Final1716295251.mov
from react-native-vision-camera-face-detector.
Okay I definitely need to update the documentation on frame.orientation
as a lot of people seem to misunderstand what it actually means and I get asked about this almost daily.
- Orientation is not fully implemented. I'm trying to raise money so I can implement orientation here: mrousavy/react-native-vision-camera#1891
- The orientation of the
Frame
is probably not what most people think under orientation. Maybe I can find a different name for it. But it is always the way you need to read the Frame to make sure it is up-right. That's because Camera sensors are often in landscape orientation and that means you need to rotate the Frame by 90deg to display it up-right. This is handled already for preview photo and video, but in Frame Processors you need to do it yourself - if I would always rotate the frame to be 90deg upright position, that would introduce a huge latency and performance bottleneck in the pipeline. So instead, ML models work by internally just reading the Frame rotated. That way I don't have to actually rotate any buffers around and it's much faster.
Hope this explains why it stays the same.
from react-native-vision-camera-face-detector.
can you add a screenshot or a video showing what you mean with "detections appear erratic"?
from react-native-vision-camera-face-detector.
Okay, can you please check if frame.orientation
change when you rotate your device?
Currently I’m using frame’s orientation to determine MLKit orientation.
from react-native-vision-camera-face-detector.
My app was locked in portrait mode so I just unlocked it. Oddly frame.orientation
still never changes. Orientation change does get captured with expo-screen-orientation
though (see recording below). So I guess it is a react-native-vision-camera
issue? It is possible that expo
or the screen orientation lib interferes with frame.orientation
@mrousavy?
What would be awesome (and maybe the simpler fix) would be to be able to manually pass the screen orientation to detectFaces
. May also enable to lock the UI in portrait mode while still handling frames for vision computing in their actual directions like it's done on instagram, tiktok, etc. What do you think?
bug2.mov
from react-native-vision-camera-face-detector.
I'll do some tests and try to implement this feature ASAP. The problem is I'm busy with another project right now so I can't say when I'll have time to do this.
Anyway, if you want to try it yourself: orientation is handled on native side in these lines:
react-native-vision-camera-face-detector/ios/VisionCameraFaceDetector.swift
Lines 220 to 231 in 4d3629e
a good start point is to set fixed values on native side and check if it works:
// ios
let image = VisionImage(buffer: frame.buffer)
image.orientation = .up //or .left or .right etc....
// android
val image = InputImage.fromMediaImage(frame.image, 0) // or 90, 180, etc...
maybe we can add the orientation
prop on config so when orientation changes it will recreate face detector:
const orientation = someOrientationHook()
const { detectFaces } = useFaceDetector( {
... other props,
orientation
} )
Or maybe get device's orientation on native side instead?
from react-native-vision-camera-face-detector.
Related Issues (20)
- Plugin with id 'kotlin-android' not found.[BUG] HOT 1
- [FEAT] Is this available ? HOT 1
- [BUG] App not working (crashing?!) on release build HOT 23
- [BUG] I cant use format prop in new release. It throw error. HOT 1
- [BUG] Release builds failing on Android with VisionCamera v4 HOT 4
- Frame Processor: Not working when creating release build HOT 6
- [Version 1.6.1 is notworking, need to new API] HOT 10
- react-native-vision-camera-face-detector:compileDebugKotlin HOT 4
- [QUESTION 🤔] Newer versions contain less features (no fps, no return converted frame) HOT 1
- npx expo run:ios gives could not find module 'ExpoModulesCore' for target 'x86_64-apple-ios-simulator' HOT 1
- smilingProbability not included HOT 1
- [BUG 🐛] java.lang.IllegalStateException: maxImages (6) has already been acquired, call #close before acquiring more. HOT 1
- [BUG 🐛] IOS inconsistent face detection
- How do I use image path instead of frame object? HOT 1
- [BUG 🐛] CMake Error in CMakeLists.txt Imported target "react-native-worklets-core::rnworklets" includes non-existent path HOT 1
- Question: How to represent a face position on a 2D canvas?
- [BUG 🐛] CocoaPods required a higher minimum deployment target HOT 1
- [BUG 🐛] [Android] faceDetectionCallback not detecting face on release mode HOT 1
- [BUG 🐛] IOS not working HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from react-native-vision-camera-face-detector.