FPS Support + Higher Resolutions

[saki4510t]

FPS Support + Higher Resolutions

[saki4510t]

Hi,

FPS limitation will be came from following reasons;

1. USB bandwidth limitation
2. MJPEG recording speed
3. memory access speed in the device
4. rendering speed

We can do nothing for reason1 but if you can use some devices that have USB3.0 port, it will become better.
For reason 2, current project use libjpeg for MJPEG decoding and you can replace it much faster libraries like libjpeg-turbo or commercial ones. libjpeg-turbo is directly replaceable with libjpeg and x1.5-x2 faster on best case(depends on the content of images, device performance etc.).
For reason 3&4, that is hardware limitation of your device. You will be able to use latest much faster device.
Additionally, there are other limitation for 3&4. Current library renders video images by transferring video images via Canvas like when you render images using SurfaceView.
As you can see in UVCPreview.cpp, MJPEG frames are converted to yuyv format and then converted rgbx and then render now. As I tested, the fast way is converting MJPEG to yuyv and pass them as texture of OpenGL|ES and render them with color space conversion on GPU(with fragment shader).
If you use TextureView/SurfaceTexture for rendering, calling SurfaceTexture#setDefaultBufferSize with native images size will be faster little.

saki

[nathantsoi]

Interesting, I didn't realize changing the resolution/frame rate is possible. Could you share your code @Katana1234?

[Katana1234]

@saki4510t
thank you for your reply :)
USB bandwidth limitation and memory access speed in the device are not the problem.
Other available apps on google play are showing the preview smooth at higher res and fps.
so it might be the MJPEG recording speed or the rendering speed.
i will try to replace the libjpeg library...
is there a reason why you've chosen the libjpeg lib instead of libjpeg-turbo?

@nathantsoi
Changing the resolution as well as the format (mjpeg, yuyv) is possible in all projects of this repository.
Changing the fps needs to edit and recompile the libs.
You have to add a new parameter for fps in the "setPreviewSize"-function of the UVCCamera lib.
Then recompile the libs and edit the native function calls in your android project.
Sharing the complete code might not be the best way for your, because i've used this project as a base for a very very custom purpose (Backup Camera App for my car).
So my code is overloaded with unneeded functions, controls and objects to show car patterns and such ;-)
@saki4510t
if you are interessted, i can send you my edited lib code, so you can provide fps support in your next demo ;)

[saki4510t]

Hi,

Actually I already implemented FPS selection function for my apps and I will add it to this project. Unfortunately I have little time to do and will takes a time.
saki.

[dwztommy123]

Hi,
please send to me your edited lib code,when i use resolution higher than 640x480,camea not work.
thanks!

[saki4510t]

Hi,

but when i choose a resolution higher than 640x480 (e.g. 1280x720@60fps@mjpeg) the preview starts lagging.

Well if you can see only several frames per seconds, I'm not sure how you change the sample project, I assume it is not a performance of mjpeg decoding issue and most possible reason will be that your camera does not support higher resolution with mjpeg mode.
Do you see logCat message that app successfully negotiated the resolution with mjpeg mode? You can seen like this on logCat, frameSize=(1280,720)@MJPEG or frameSize=(1280,720)@YUYV.
Just passing FRAME_FORMAT_MJPEG is not enough because most sample projects fallback to YUV mode if the camera does not support the resolution with specific mode.
Could you provide device descriptor of you camera? It is OK that get it with PC or set LOCAL_DEBUG macro to 1 in UVCCamera.cpp and build & run & get it from logCat.
saki

[Katana1234]

@saki4510t
Hi saki,

the camera definitely supports mjpeg mode with higher framerates.
sadly i've already mounted my camera in my car as a backup camera so i can't give you the logCat, but...
My camera supports e.g.:
1929x1080 @ 30fps @ mjpeg
1280x720 @ 60fps @ mjpeg
640x480 @ 120fps @ mjpeg
and a lot more in mjpeg as well as yuv mode, but with much less framerates.

When connecting the cam to Windows i can setup the resolution as well as the mode in VLC. And it works perfect and smooth.
When connecting the cam to my Nexus 7 while using the app "CameraFi" 1280x720 gives me 60fps with a little bit of lagging sometimes ;) i don't know which lib this app is using.

i've also extended the libs as well as the app to support dynamic frame rates and also created a "choose resolution" dialog, which shows the available resolutions, framerates and modes given by the camera.
all these functions work perfectly but i still guess the lib might be to slow.
when i choose 640x480 @120FPS @Mjpeg the camera preview also works good, but it also laggs. so i guess, some component of this app-lib combination is not made for higher datarates or is to slow for mjpeg.

as i've already mentioned, i already mounted my camera in my car, and i'm pretty happy and thankful for your work :) i'm using 640x480 @30fps yuv because all higher settings start lagging. but for me, it's not a problem :)

if you want to, i can close this issue, because i can't test anymore...

[osburd]

Hello saki
I would like to use your code with my HD UVC cam. Unfortunately, I facing huge lag described in this thread at 1920x1080 30fps preview (vga looks good).
The thread was closed 1.5 years ago and I hope, somebody solved it? If yes, can you advice me please or refer to somebody who have a solution?
Thanks in advance.
Alex

[rabdulmanan]

Hi,
I m facing same issue as describe in this thread. when i choose a resolution higher than 640x480 e.g. 1920x1080 30fps mjpeg preview start lagging and drop fps to 10-5fps. Is there any solution to solve the problem.