How do I pass my own acquired CMSampleBufferRef objects?

[card-io]

How do I pass my own acquired CMSampleBufferRef objects?

[dgoldman-pdx]

@benshepherd I'm not entirely clear on what you're trying to do. At a high level, could you explain which parts of our library you're trying to replace with your own code, and why?

[benshepherd]

Sure - I already have a "document capture" pipeline with several features including a UI. My pipeline does both GPU and non-GPU-based processing. I'd prefer to grab CMSampleBuffers from my pipeline, pass them to CardIO for credit card data extraction, and get the results back. It looked to me as if the best way to do this would be to manually create CardIOVideoFrame objects and use your existing plumbing from there.

[dgoldman-pdx]

I guess that might work, if you can figure out where to feed in the images and get back the results...

You'll need to configure your session with the same resolution, pixel format, etc that we use. Off the top of my head, I think that all happens inside CardIOVideoStream.mm.

In general, I guess you'll be pretending to be a CardIOVideoStream...?

[benshepherd]

Yes - that was my thought. However, when I pull in libCardIO.a and the CardIOVideoStream.h, the compiler doesn't like it. Maybe there's some Xcode trick to make CardIOVideoStream's methods visible? CardIOView works fine.

[dgoldman-pdx]

Ah, you're trying to use our official libCardIO.a. Yeah, most of its internal symbols are not public -- that's the result of a lot of work to help out apps that already have their own copy of the OpenCV library and were running into symbol conflicts with the OpenCV inside libCardIO.a.

You'll probably need to directly pull in the code you need from this source repo, rather than getting away with the official library.

Or, hmm...

If you can clone this source repo and get the library to build, then you should be able to make the necessary symbols public by adding them to this file: https://github.com/card-io/card.io-iOS-source/blob/master/SupportFiles/CardIO_exported_symbols.txt

[benshepherd]

Great idea - thanks! I'll give it a shot.

[benshepherd]

I added CardIOVideoFrame to the exported symbols and re-built from Xcode. The new libCardIO.a has the symbols present (confirmed via the 'nm' command), but my Xcode project still doesn't recognize it. I've included the headers in my project and #import "CardIOVideoFrame.h". Strange.

[dgoldman-pdx]

@benshepherd be sure to include both _OBJC_CLASS_$_CardIOVideoFrame and _OBJC_METACLASS_$_CardIOVideoFrame as well as the specific method names.

This being a .mm file might also be complicating things, I suppose.

Since you're building from Xcode anyhow, and if you won't have any conflicts with OpenCV symbols, another approach might be simply to remove the linker settings that are hiding all non-exported symbols. If you can't figure out what and where those settings are in the project's build settings, let me know and I'll help you dig them out.

[benshepherd]

The main issue turned out to be the #if statements at the beginning of the CardIOVideoFrame.h !! That was the only reason I couldn't see the symbols after adding them to the exports file. My CMSampleBuffers are 1280x720, so I will need to downscale? CardIOVideoStream defaults to 640x480, but it looks like you support other resolutions (1280x720, etc). I have it running, but I'm getting no cardInfo back from the video frame.

[dgoldman-pdx]

The main issue turned out to be the #if statements at the beginning of the CardIOVideoFrame.h !! That was the only reason I couldn't see the symbols after adding them to the exports file.

We do have a bit of a collection of #defines, don't we? Sorry about that!

My CMSampleBuffers are 1280x720, so I will need to downscale?

Ah. Tricky...

Originally, card.io was designed and released for 640x480 resolution – because that was the only iPhone camera resolution available at the time.

A couple of years ago we did some experiments with using higher camera resolutions. As a result of those experiments, some of our pipeline became, to some extent anyhow, resolution-agnostic. However, we did not carry this resolution-agnosticity all the way through to the final stages of character detection and recognition. Doing so would require a lot more changes to our code, which makes quite a few resolution and aspect-ratio assumptions. Even more challenging, it would also require us to collect an entirely new set of higher-resolution training images with which to retrain our deep-learning character-recognition models. (We're talking thousands of training images.)

During those experiments, starting with a higher-resolution image and then downsampling it later in the pipeline proved disappointing. That result is consistent with my understanding that in-camera downsampling produces much better results than post-camera downsampling, because the former can take advantage of the complex geometries within the camera's sensor which were designed specifically for this purpose.

If you can achieve better success than we did in our higher-resolution experiments, that would be fantastic, and we'd love to incorporate any such improvements into the codebase. But it won't be a quick or easy task, I'm afraid.