johndmcmaster/gst-plugin-toupcam

gst-plugin-toupcam's Issues

Review latency

Response seems slow to me right now even though I have a relatively low exposure time. Feels like some images might be stuck in the pipeline somewhere

Implement manual exposure control

Clean up readme

Looks like we have a basic idea of how to set it up. Readme has misc issues (bad steps, unclear order, etc)

Way to support selecting 1 of N cameras

So someone tried to use pyuscope and has a manual scope also attached to the same computer. Ideally have a way to select by model and/or serial number

Review controls

MVP controls:

• Exposure
• Control or at least disable auto-exposure
• Red gain
• Green gain
• Blue gain

auto_exposure can't be disabled

pipeline works briefly and then crashes

GST_DEBUG=0 GST_PLUGIN_PATH=$PWD/src/.libs/ gst-launch-1.0 toupcamsrc auto_exposure=1 ! videoconvert ! xvimagesink
registering...

(gst-launch-1.0:2542): GLib-GObject-CRITICAL **: g_object_class_install_property: assertion 'property_id > 0' failed

** (gst-launch-1.0:2542): WARNING **: gsttoupcamsrc.c:196: invalid property id 4 for "auto-exposure" of type 'GParamBoolean' in 'GstToupCamSrc'
Setting pipeline to PAUSED ...
start
calllback 1 (want 4), images now 0
calllback 1 (want 4), images now 0
calllback 1 (want 4), images now 0
calllback 1 (want 4), images now 0
calllback 1 (want 4), images now 0
calllback 1 (want 4), images now 0
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

calllback 1 (want 4), images now 0
calllback 4 (want 4), images now 1
pull new image
pull image ok, total = 1, resolution = 5440 x 3648
flag 3, seq 3, us 224150
calllback 4 (want 4), images now 2

...

calllback 4 (want 4), images now 14
pull new image
pull image ok, total = 12, resolution = 5440 x 3648
flag 3, seq 15, us 2970395

ERROR: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Output window was closed
Additional debug info:
xvimagesink.c(555): gst_xv_image_sink_handle_xevents (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0
Execution ended after 0:00:04.208417058
Setting pipeline to PAUSED ...
calllback 4 (want 4), images now 15
pull new image
pull image ok, total = 13, resolution = 5440 x 3648
flag 3, seq 16, us 3273944
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Crash when using pyqt tee

TLDR: more advanced usage is causing a SIGSEGV in master. I've done some preliminary troubleshooting and discovered that its related to this section (modified slightly from master):

    gst_buffer_map (buf, &minfo, GST_MAP_WRITE);
    printf("minfo size %lu, maxsize %lu, flags 0x%08X\n", minfo.size, minfo.maxsize, minfo.flags);
    //FIXME: debugging crash
    if (minfo.size != 59535360) {
        printf("bad minfo size\n");
        exit(1);
    }

Instead of getting a 59535360 byte buffer for a whole frame, its only 4096 bytes. This only happens when using a tee and only from pyqt I think. For example, this seems to work:

gst-launch-1.0 toupcamsrc ! videoconvert ! tee name=t ! queue ! autovideosink t. ! queue ! autovideosink

Python test case: https://gist.github.com/JohnDMcMaster/2d8d10df931b51ec4fee75429f341707

# Crash
python3 buff_crash.py gst-toupcamsrc
# ok
python3 buff_crash.py gst-v4l2src
# ok
python3 buff_crash.py gst-toupcamsrc

Investigate E3ISPM25000KPA stability

We've observed several issues with this camera with the official CD + ToupView app (SDK 53.21522.20221011). Ex: autoexposure broke. Similarly pyuscope was relatively quickly crashing, especially at higher frame rates.

A few theories:

• High MP camera => stressing software. Evidence: lowering resolution reduces but does not eliminate crashes
• Exposure related => long (auto) exposure times out. Evidence: flickering light on / off encourages crash

We also have a newer SDK to test sent directly from ToupTek (#27)

Investigate failed start

$ GST_DEBUG=3,toupcamsrc:5,GST_TOUPCAMSRC_INFO=Y gst-launch-1.0 toupcamsrc
0:00:00.036867304 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:168:gst_toupcam_src_class_init: Using gst_toupcam_src_fill
Setting pipeline to PAUSED ...
0:00:00.037012792 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:780:gst_toupcam_src_start:<toupcamsrc0> start
0:00:00.037028150 275779 0x558db2ee2f80 INFO              toupcamsrc gsttoupcamsrc.c:787:gst_toupcam_src_start:<toupcamsrc0> ToupCam Library Ver 53.21907.20221218
0:00:00.037431023 275779 0x558db2ee2f80 INFO              toupcamsrc gsttoupcamsrc.c:792:gst_toupcam_src_start:<toupcamsrc0> Found 1 devices
0:00:00.037439815 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:798:gst_toupcam_src_start:<toupcamsrc0> Toupcam_Open
0:00:00.372837037 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:879:gst_toupcam_src_start:<toupcamsrc0> setup image mode: regular
0:00:00.374345781 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:924:gst_toupcam_src_start:<toupcamsrc0> Image 4928 w x 4928 h, in 3 bytes / pix => 72855552 bytes (72.9 MB), out 3 bytes / pix => 72855552 bytes (72.9 MB)
0:00:00.519099712 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:667:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=1) begin, want 4
0:00:00.519141691 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:674:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=1) end, images now 0
0:00:00.553884882 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:667:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=1) begin, want 4
0:00:00.553932768 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:674:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=1) end, images now 0
0:00:00.647302287 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:667:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=1) begin, want 4
0:00:00.647358596 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:674:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=1) end, images now 0
0:00:00.653176631 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:667:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=1) begin, want 4
0:00:00.653231878 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:674:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=1) end, images now 0
0:00:00.665004318 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:942:gst_toupcam_src_start:<toupcamsrc0> gst_toupcam_src_start(): ok
Pipeline is live and does not need PREROLL ...
0:00:00.665503885 275779 0x558db2e18aa0 FIXME                default gstutils.c:3980:gst_pad_create_stream_id_internal:<toupcamsrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
Setting pipeline to PLAYING ...
0:00:00.665853449 275779 0x558db2e18aa0 WARN              video-info video-info.c:727:gst_video_info_to_caps: invalid matrix 0 for RGB format, using RGB
0:00:00.665952418 275779 0x558db2e18aa0 INFO              toupcamsrc gsttoupcamsrc.c:1030:gst_toupcam_src_set_caps:<toupcamsrc0> The caps being set are video/x-raw, format=(string)RGB, width=(int)4928, height=(int)4928, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)0:1:0:0, framerate=(fraction)0/1
New clock: GstSystemClock
0:00:00.666077443 275779 0x558db2e18aa0 DEBUG             toupcamsrc gsttoupcamsrc.c:1291:gst_toupcam_src_fill:<toupcamsrc0> gst_toupcam_src_fill()
0:00:00.666095312 275779 0x558db2e18aa0 DEBUG             toupcamsrc gsttoupcamsrc.c:1303:gst_toupcam_src_fill:<toupcamsrc0>  
0:00:00.666106437 275779 0x558db2e18aa0 DEBUG             toupcamsrc gsttoupcamsrc.c:1304:gst_toupcam_src_fill:<toupcamsrc0> waiting for new image
0:00:00.801306710 275779 0x7fdbec002320 DEBUG             toupcamsrc gsttoupcamsrc.c:667:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=1) begin, want 4
0:00:00.801372474 275779 0x7fdbec002320 DEBUG             toupcamsrc gsttoupcamsrc.c:674:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=1) end, images now 0
0:00:00.899989996 275779 0x7fdbf40038d0 DEBUG             toupcamsrc gsttoupcamsrc.c:667:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=4) begin, want 4
0:00:00.900015579 275779 0x7fdbf40038d0 DEBUG             toupcamsrc gsttoupcamsrc.c:674:sdk_callback_PullMode:<toupcamsrc0> sdk_callback_PullMode(nEvent=4) end, images now 1
0:00:00.900070779 275779 0x558db2e18aa0 DEBUG             toupcamsrc gsttoupcamsrc.c:1171:pull_decode_frame:<toupcamsrc0> minfo size 72855552, maxsize 72855559, flags 0x00000002
0:00:00.900081216 275779 0x558db2e18aa0 DEBUG             toupcamsrc gsttoupcamsrc.c:1240:pull_decode_frame:<toupcamsrc0> pulling x8 image
0:00:00.927148819 275779 0x558db2e18aa0 DEBUG             toupcamsrc gsttoupcamsrc.c:1255:pull_decode_frame:<toupcamsrc0> pull image ok, total = 1, resolution = 4928 x 4928
0:00:00.927164442 275779 0x558db2e18aa0 DEBUG             toupcamsrc gsttoupcamsrc.c:1258:pull_decode_frame:<toupcamsrc0> flag 3, seq 1, us 128917
0:00:00.930580316 275779 0x558db2e18aa0 WARN                 basesrc gstbasesrc.c:3072:gst_base_src_loop:<toupcamsrc0> error: Internal data stream error.
0:00:00.930589100 275779 0x558db2e18aa0 WARN                 basesrc gstbasesrc.c:3072:gst_base_src_loop:<toupcamsrc0> error: streaming stopped, reason not-linked (-1)
ERROR: from element /GstPipeline:pipeline0/GstToupCamSrc:toupcamsrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/GstToupCamSrc:toupcamsrc0:
streaming stopped, reason not-linked (-1)
Execution ended after 0:00:00.264674767
Setting pipeline to NULL ...
0:00:00.930798068 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:960:gst_toupcam_src_stop:<toupcamsrc0> gst_toupcam_src_stop()
Freeing pipeline ...
0:00:00.988219855 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:643:gst_toupcam_src_dispose:<toupcamsrc0> dispose
0:00:00.988243797 275779 0x558db2ee2f80 DEBUG             toupcamsrc gsttoupcamsrc.c:657:gst_toupcam_src_finalize:<toupcamsrc0> finalize

Maybe somewhat related to #28

Streamline 12/16 vs 8 bit image pipeline

Currently in order to switch between 8 and 16 bit output image formats you need to switch a compile flag. This is due to the need to switch classes during class_init() which exposes static caps. However, it looks like the actual caps might be set via get_caps() which constructs it via gst_video_format_get_info(). Understand the relationship between these two and expose properties instead of compile flags if possible.

Negotiate resolution via caps

We should ideally expose resolutions to negotiate via caps. Right now you can manually set an esize property, but its not the intuitive / expected way to do things

Stream images

Current code is PoC where frames are loaded at initialization

Images are ugly

Looks blue to me. Maybe some sort of interlacing as well. Perhaps its giving 12 bit images and displaying as 8 bit?

Investigate branded SDK support (Switftcam, nncam, etc)

Looks like Switcam SC series are Touptek cameras. However, they don't see to open with my version of toupcamlite. Investigate the differences and see if its worth supporting.

DUT: SWITFTCAM SC1803R

$ lsusb
Bus 001 Device 012: ID 0547:c253 Anchor Chips, Inc. SC1803R

Noting:

$ cat /etc/udev/rules.d/99-toupcam.rules
....
SUBSYSTEM=="usb", ATTRS{idVendor}=="0547", MODE="0666"

Digging into Swift SDK:

$ cat inc/swiftcam.h
#ifndef __swiftcam_h__
#define __swiftcam_h__

/* Version: 49.18830.20210423 */
/*
   Platform & Architecture:
       (1) Win32:
              (a) x86: XP SP3 or above; CPU supports SSE2 instruction set or above
              (b) x64: Win7 or above
              (c) arm: Win10 or above
              (d) arm64: Win10 or above
       (2) WinRT: x86, x64, arm, arm64; Win10 or above
       (3) macOS: universal (x64 + x86); macOS 10.10 or above
       (4) Linux: kernel 2.6.27 or above; GLIBC 2.17 or above
              (a) x86: CPU supports SSE3 instruction set or above; built by gcc 5.4.0
              (b) x64: built by gcc 5.4.0
              (c) armel: built by toolchain arm-linux-gnueabi (version 4.9.2)
              (d) armhf: built by toolchain arm-linux-gnueabihf (version 4.9.2)
              (e) arm64: built by toolchain aarch64-linux-gnu (version 4.9.2)
       (5) Android: arm, arm64, x86, x64; built by android-ndk-r18b; __ANDROID_API__ = 23
*/
...
#define SWIFTCAM_FLAG_CMOS                0x00000001  /* cmos sensor */
#define SWIFTCAM_FLAG_CCD_PROGRESSIVE     0x00000002  /* progressive ccd sensor */
#define SWIFTCAM_FLAG_CCD_INTERLACED      0x00000004  /* interlaced ccd sensor */
...

vs touptek lib I have:

$ cat inc/toupcam.h
#ifndef __toupcam_h__
#define __toupcam_h__

/* Version: 46.17309.2020.0616 */
/*
   Platform & Architecture:
       (1) Win32:
              (a) x86: XP SP3 or above; CPU supports SSE2 instruction set or above
              (b) x64: Win7 or above
              (c) arm: Win10 or above
              (d) arm64: Win10 or above
       (2) WinRT: x86, x64, arm, arm64; Win10 or above
       (3) macOS: universal (x64 + x86); macOS 10.10 or above
       (4) Linux: kernel 2.6.27 or above
              (a) x86: CPU supports SSE3 instruction set or above; GLIBC 2.8 or above
              (b) x64: GLIBC 2.14 or above
              (c) armel: GLIBC 2.17 or above; built by toolchain arm-linux-gnueabi (version 4.9.2)
              (d) armhf: GLIBC 2.17 or above; built by toolchain arm-linux-gnueabihf (version 4.9.2)
              (e) arm64: GLIBC 2.17 or above; built by toolchain aarch64-linux-gnu (version 4.9.2)
       (5) Android: arm, arm64, x86, x64; built by android-ndk-r18b; __ANDROID_API__ = 23
*/
/*
    doc:
       (1) en.html, English
       (2) hans.html, Simplified Chinese
*/
...
#define TOUPCAM_FLAG_CMOS                0x00000001  /* cmos sensor */
#define TOUPCAM_FLAG_CCD_PROGRESSIVE     0x00000002  /* progressive ccd sensor */
#define TOUPCAM_FLAG_CCD_INTERLACED      0x00000004  /* interlaced ccd sensor */
...

Add properties for auto exposure knobs

Target and what not

Understand balance

TLDR: what is the proper way to control balance? Ideally using gain registers and not post processing

Summary:

• black balance works. 0 => normal, 255 => drop channel. One channel each for R, G, and B
• Added black balance controls
• I can Toupcam_get_TempTint() but not Toupcam_put_TempTint(). However Toupcam_AwbOnePush() can adjust settings
• None of the RGB family functions is working for me
• Also have hue etc controls, but not sure if those are the right things

Plugin fails to run on newer gstreamer

Says plugin gave internal error or something like that

12 bit support

I have a 12 bit camera. Make sure this gets through the full pipeline

Test pattern generator

As far as I can tell libtoupcam doesn't have a test pattern generator, but these are very useful for quick testing (ex: with pyuscope)

Support gain control

Allocate frame buffer to needed size

A max size (25 MP w/ 64 bpp) is hard coded in. its probably fine but is a little dirty

Cut release

Evidently there are no official releases. Let's cut an official release with supported version numbers as part of recent SDK testing

Optimize touptek GBRG12 to gstreamer ARGB64 conversion

Take a look at proof of concept GBRG12_to_ARGB64_x4() function and see if it can be made faster.

Other: we should also check Toupcam_get_RawFormat() and select and/or assert conversion based on FourCC value

Provide a way to enumerate camera resolutions

Currently you need to use the GUI to find esize + resolution pairing which isn't ideal

A few options:

• Ideal: report all possible resolution to gstreamer and negotiate things. But that's a lot more complicated
• Add some properties, even if not proper
• Print at startup (with some verbose flag?)

gst_video_info_to_caps: invalid matrix 0 for RGB format, using RGB

Investigate GstVideoInfo usage and if we should do something more explicit

Example:

0:00:00.837126692 262591 0x7fae80006300 WARN              video-info video-info.c:727:gst_video_info_to_caps: invalid matrix 0 for RGB format, using RGB

Frame count off by one error

gst-launch-1.0 toupcamsrc num-buffers=1 ! filesink location=test1.bin
gst-launch-1.0 toupcamsrc num-buffers=2 ! filesink location=test2.bin
gst-launch-1.0 toupcamsrc num-buffers=3 ! filesink location=test3.bin

Gives

-rw-rw-r-- 1 mcmaster mcmaster    0 Aug 25 02:01 test1.bin
-rw-rw-r-- 1 mcmaster mcmaster  57M Aug 25 02:01 test2.bin
-rw-rw-r-- 1 mcmaster mcmaster 114M Aug 25 02:01 test3.bin

Consider source plugins

Consider porting kernel code and/or a few of the user space projects to gstreamer plugins. Either directly or through an intermediate native library. Cameras today with at least some coverage:

• MU300: kernel
• MU800: kernel (mainline), user
• MD1800: kernel?
• MD1900: user

Plugin fails to load on newer gstreamer

tested: 1.8.3

but now ubuntu 20: 1.16.2

TLDR: its probably this issue http://gstreamer-devel.966125.n4.nabble.com/Plugin-loading-fails-with-Gstreamer-1-14-0-td4686497.html

Test 53.21907.20221217

Ideally we should support these version:

• 53.21907.20221217: touptek email
• 53.21522.20221011: E3ISPM25000KPA CD
• 50.19728.20211022: latest public version

Cut and bump version if this looks ok. Add supported SDK version(s) to release notes