Comments (14)
@SasLuca I was able to get my integrated webcam working with this change, maybe you'll have some luck?
0001-camera-Support-CoreMedia-devices-with-NV12-pixel-for.patch
I am seeing the bad access as well. I can repro by running testcamera
on my laptop and selecting my iPhone camera as the source.
from sdl.
It's probably just an oversight in CoreMediaFormatToSDL(), please feel free to extend it and submit a PR.
from sdl.
I did some light research on this and the issue isn't so much with CoreMediaFormatToSDL
but with how formats are queried and which ones are supported.
The SDL camera code for coremedia uses CMFormatDescriptionGetMediaSubType
but it seems that in order to get all formats a different method must be used as per this SO post, and even then it is not clear if the formats will be supported by SDL.
I will implement the method for getting formats that is suggested in that SO post and see if the formats will then match. If they don't however I am not quite sure how to proceed, would it mean that SDL needs to add support for more formats?
from sdl.
I will implement the method for getting formats that is suggested in that SO post and see if the formats will then match. If they don't however I am not quite sure how to proceed, would it mean that SDL needs to add support for more formats?
It's possible, yes, but it's likely that SDL already supports the common formats used by the camera.
from sdl.
It seems like that SO post about CoreMediaIO was a bad lead, as CoreMediaIO is for macos only.
I instead tried to print all the formats the cameras on my iPhone 13 support, it seems like they are part of another enum than what SDL uses. SDL is looking for formats in the CMPixelFormatType
, but that list is apparently not exhaustive. A longer list of formats can be found in CVPixelBuffer.h and the formats in this list do indeed match the ones the cameras support on my iPhone 13.
In particular x420, 420v and 420f seem very common. These format codes translate to the following formats:
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange = '420v'
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange = '420f'
kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange = 'x420'
I think I could modify the SDL code to use this header and values instead, but from there on I am sadly not quite knowledgeable. Does SDL support these formats? Is there a way for SDL to support this formats?
Not sure what the implications are here and what needs to be done so any help would be appreciated, I would really love to use the Camera API for my mobile game.
from sdl.
Yes, SDL does support these formats. The difference between kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange and kCVPixelFormatType_420YpCbCr8BiPlanarFullRange is not represented in the SDL pixel format, but instead in other metadata for the surface. I'll have to look at this later, but for now you can just return SDL_PIXELFORMAT_NV12 for both of them and it should work.
from sdl.
Awesome, I will modify the code on my side for now and see if I can get the camera stuff to work. Thanks a lot.
from sdl.
You're welcome!
from sdl.
This is related to #9610
I was investigating this earlier today and found that adding CASE(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, SDL_PIXELFORMAT_NV12);
to CoreMediaFormatToSDL
does not work as expected. In my case, the device is recognized, but the output frames are not displayed correctly.
While debugging I found that the built-in camera on my M1 Macbook Air reports the pixel format kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange
but actually outputs frames as kCMPixelFormat_422YpCbCr8_yuvs
:
In COREMEDIA_OpenDevice
:
(lldb) p (uint32_t)CMFormatDescriptionGetMediaSubType(spec_format.formatDescription)
(uint32_t) 875704438 // '420v'
In COREMEDIA_AcquireFrame
:
(lldb) p (uint32_t)CVPixelBufferGetPixelFormatType(image)
(uint32_t) 846624121 // '2vuy'
I had to add CASE(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, SDL_PIXELFORMAT_UYVY)
to get testcamera
to work, which doesn't seem right at all.
from sdl.
@slouken I tried your suggestion of using SDL_PIXELFORMAT_NV12
for those camera formats, and while it did work at getting past camera device collection, during rendering I would be getting EXC_BAD_ACCESS
thrown from this line in COREMEDIA_AcquireFrame.
I then tried @hkva's suggestion of mapping kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange
to SDL_PIXELFORMAT_UYVY
. This still resulted in an EXC_BAD_ACCESS
, but only once, and after that it would actually render! Sadly the rendering was corrupted, as shown in this image.
I then tried updating SDL, since I last got it from the main branch 2 weeks ago or so, and now I just get EXC_BAD_ACCESS
on that same line non stop regardless of the format.
Any help regarding what I could try next to debug or fix this would be appreciated, I really want to use the Camera API so I would like to contribute in any way I can, tho sadly I am not very well educated in this particular area.
Some logs that stand out to me and I'll investigate now are INFO: CAMERA: App wanted [(0x0) fmt=SDL_PIXELFORMAT_UNKNOWN interval=1000/1], chose [(4032x3024) fmt=SDL_PIXELFORMAT_UYVY interval=1/1]
and Thread Performance Checker: -[AVCaptureSession startRunning] should be called from background thread. Calling it on the main thread can lead to UI unresponsiveness
Here is a picture of the EXC_BAD_ACCESS
I am getting. I am also logging with DEBUG_CAMERA
on.
from sdl.
Ill try your fork and let you know if it works, thanks a lot.
from sdl.
@hkva tried the wrong branch initially, but I did try 9610-coremedia-camera-nv12 now and still getting bad accesses in that place sadly.
from sdl.
Fixed, thanks!
from sdl.
Awesome, will be updating and trying it again soon!
from sdl.
Related Issues (20)
- 10,000th issue 🥳 HOT 1
- [Camera] CoreMedia: Incorrect pixel format used for USB HDMI Video grabber (YUYV 4:2:2) HOT 3
- SDL2 SDL_Keycode returns characters in current keyboard layout HOT 2
- SDL3 doesn't statically link without SDL_STATIC_PIC flag on Ubuntu 24.04
- SDL3 static link fail when making a shared object without SDL_STATIC_PIC HOT 4
- [Regression (again)] Metal YUV rendering broken on Intel MacOS HOT 15
- SDL_ReadU32LE vs SDL_SwapLE32: location of LE suffix is inconsistent HOT 2
- There is no `SDL_ReadS8` HOT 1
- [email protected]
- `SDL_SetRelativeMouseMode` in `SDL_EVENT_MOUSE_BUTTON_DOWN` blocks until the button is released HOT 1
- SDL_JOYSTICK_DISABLED breaks Android build with a linker error HOT 3
- Server side decorations broken on SDL2 Wayland HOT 2
- macOS 10.14 SDK Compatibility HOT 1
- [Camera]: SDL_GetCameraPermissionState returns 0 if pipewire backend is being used HOT 1
- [SDL3] Is SDL_PROP_TEXTURE_CREATE_D3D11_TEXTURE_POINTER not implemented? HOT 1
- Linux - Installing SDL2 .so file using CMake HOT 1
- Source engine toGL games crash when not using x11 HOT 2
- Crash with SDL_EVENT_GAMEPAD_REMOVED [[repro inside]] HOT 17
- Unnecessary CMake re-configures triggered by git history changes HOT 3
- SDL3 Linux Wayland - SDL_SetWindowFullscreen incorrect size HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sdl.