svtlabs/screencapturekit-rs

Safe rust wrapper around Apple's ScreenCaptureKit

screencapturekit-rs's Issues

Consider a more permissive license

Hi, I was hoping to use this in my MIT licensed project, but unfortunately the LGPL license would cause me to have to license the final binary I distribute as LGPL, if I statically link. Would you consider adopting a license that doesn't force other projects to use the same license/is non-viral?

One approach you could consider is dual licensing as MIT and Apache 2.0, just like rustc itself and most of the rest of the rust ecosystem.

Use rustdoc

In order to have the code generate a better docs page, the api needs to be documented better.

Make sure that rustdoc? can generate a nice looking docs page.

Audio stream support

I found this library while finding the ScreenCaptureKit library. I want to record audio but i guess, add_stream_output is hard-corded to 0(SCStreamOutputType::Screen).
https://github.com/svtlabs/screencapturekit-rs/blob/f2327fc1206fc7f32918aed3fea2de0775b8ac36/screencapturekit-sys/src/stream.rs#L76C1-L77C1

How do I read the image buffer?

Hi, me again 😓

I'm working on a screen capture library and would love to get this crate working within it. Having some difficulty getting to the ImageBuffer that I can use. Does the image buffer need to be exposed from the crate itself yet?

Here's my current snippet where I'm stuck:

use screencapturekit::{
    sc_content_filter::{InitParams, SCContentFilter},
    sc_error_handler,
    sc_output_handler::{CMSampleBuffer, SCStreamOutputType, StreamOutput},
    sc_shareable_content::SCShareableContent,
    sc_stream::SCStream,
    sc_stream_configuration::SCStreamConfiguration,
    sc_sys::{CMSampleBufferGetImageBuffer, CMSampleBufferRef, SCFrameStatus},
};

use core_graphics::display::CGMainDisplayID;

struct ConsoleErrorHandler;

impl sc_error_handler::StreamErrorHandler for ConsoleErrorHandler {
    fn on_error(&self) {
        println!("Error!");
    }
}

struct OutputHandler {}

impl StreamOutput for OutputHandler {
    fn did_output_sample_buffer(&self, sample: CMSampleBuffer, of_type: SCStreamOutputType) {
        match of_type {
            SCStreamOutputType::Screen => {
                let frame_status = sample.frame_status;

                match frame_status {
                    SCFrameStatus::Idle => {
                        return;
                    }
                    _ => {
                        let ptr = sample.ptr;
                        println!("Id<CMSampleBufferRef>: {:?}", ptr);

                        let timestamp = ptr.get_presentation_timestamp().value;
                        println!("Timestamp: {}", timestamp);

                        // 🤔 how do I get the ImageBuffer from here?
                    }
                }
            }
            SCStreamOutputType::Audio => {
                // TODO: handle audio
            }
        }
    }
}

pub fn main() {
    let content = SCShareableContent::current();
    let displays = content.displays;

    let main_display_id = unsafe { CGMainDisplayID() };
    let main_display = displays
        .iter()
        .find(|display| display.display_id == main_display_id)
        .unwrap_or_else(|| {
            panic!("Main display not found");
        });

    let width = main_display.width;
    let height = main_display.height;

    // Setup screencapturekit
    let params = InitParams::Display(main_display.clone());
    let filter = SCContentFilter::new(params);

    let stream_config = SCStreamConfiguration {
        shows_cursor: true,
        width,
        height,
        ..Default::default()
    };

    let error_handler = ConsoleErrorHandler;
    let mut stream = SCStream::new(filter, stream_config, error_handler);
    let output_handler = OutputHandler {};
    stream.add_output(output_handler);

    stream.start_capture();
    println!("Capture started. Press Enter to stop.");

    let mut input = String::new();
    std::io::stdin().read_line(&mut input).unwrap();

    stream.stop_capture();
    println!("Capture stopped.");
}

And my cargo.toml points to the main branch of this repo.

[target.'cfg(target_os = "macos")'.dependencies]
tao-core-video-sys = "0.2.0"
core-graphics = "0.23.1"
screencapturekit = { git = "https://github.com/svtlabs/screencapturekit-rs", branch = "main" }

CMSampleBuffer or CVPixelBufferRefs showing up empty

Hi, thank you for the great work on this package! 🙏

Wondering if you can point me in the right direction: I'm trying to save a recording of my screen to PNG files, but the CVPixelBuffer's base_address I receive seems to be empty. Should this be handled differently?

The height, width and bytes_per_row are all correct + the menubar indicator on macOS also correctly shows me the display being shared so I'm unsure why this is happening.

use screencapturekit::{
    sc_content_filter::{InitParams, SCContentFilter},
    sc_error_handler,
    sc_output_handler::StreamOutput,
    sc_shareable_content::SCShareableContent,
    sc_stream::{CMSampleBuffer, SCStream},
    sc_stream_configuration::SCStreamConfiguration,
};

use core_video_sys::{
    CVPixelBufferGetBaseAddress, CVPixelBufferGetBytesPerRow, CVPixelBufferGetHeight,
    CVPixelBufferGetWidth, CVPixelBufferRef,
};

struct ConsoleErrorHandler;

impl sc_error_handler::StreamErrorHandler for ConsoleErrorHandler {
    fn on_error(&self) {
        println!("Error!");
    }
}

struct OutputHandler;

impl StreamOutput for OutputHandler {
    fn stream_output(&self, sample: CMSampleBuffer) {
        let timestamp = sample.presentation_timestamp.value;
        let pixel_buffer =
            sample.pixel_buffer.expect("No buffer found in sample") as CVPixelBufferRef;

        // get the pixel buffer's width and height
        let width = unsafe { CVPixelBufferGetWidth(pixel_buffer) };
        let height = unsafe { CVPixelBufferGetHeight(pixel_buffer) };
        println!("size: {}x{}", width, height);

        // create an ImageBuffer from the pixel buffer data
        let bytes_per_row = unsafe { CVPixelBufferGetBytesPerRow(pixel_buffer) };
        let base_address = unsafe { CVPixelBufferGetBaseAddress(pixel_buffer) };

        // 🤔 HELP: base_address is 0x0 all the time
        println!("base_address: {:?}", base_address);

        let buffer = unsafe {
            std::slice::from_raw_parts(base_address as *const u8, bytes_per_row * height)
        };

        // 🤔 HELP: buffer is empty
        println!("buffer: {:?}", buffer);

        // let image_buffer =
        //     ImageBuffer::<Rgba<u8>, _>::from_raw(width as u32, height as u32, buffer).unwrap();

        // write the ImageBuffer to a PNG file

        // let filename = format!("output-{}.png", timestamp);
        // let file = File::create(&filename).unwrap();
        // let ref mut writer = BufWriter::new(file);
        // let encoder = image::codecs::png::PngEncoder::new(writer);
        // encoder
        //     .encode(
        //         &image_buffer,
        //         width as u32,
        //         height as u32,
        //         image::ColorType::Rgba8,
        //     )
        //     .unwrap();

        // println!("Wrote PNG file: {}", filename);
    }
}

pub fn main() -> Result<(), ()> {
    let content = SCShareableContent::current();
    let display = content.displays.first().unwrap();

    let width = display.width;
    let height = display.height;

    let params = InitParams::Display(display.to_owned());
    let filter = SCContentFilter::new(params);

    let mut stream_config = SCStreamConfiguration::default();

    stream_config.shows_cursor = true;
    stream_config.width = width;
    stream_config.height = height;
    stream_config.captures_audio = true;

    let handler = ConsoleErrorHandler;
    let mut stream = SCStream::new(filter, stream_config, handler);

    let output_handler = OutputHandler;
    stream.add_output(output_handler);

    stream.start_capture();
    println!("Capture started. Press Enter to stop.");

    let mut input = String::new();
    std::io::stdin().read_line(&mut input).unwrap();

    stream.stop_capture();
    println!("Capture stopped.");

    Ok(())
}

Trying to capture screen but not able to pass stream output type

    let display = SCShareableContent::current()
        .displays
        .into_iter()
        .next()
        .unwrap();

    let filter = SCContentFilter::new(InitParams::Display(display.clone()));
    let config = SCStreamConfiguration {
        width: display.width,
        height: display.height,

        ..SCStreamConfiguration::default()
    };
    let (tx, rx) = sync_channel(2);
    let mut stream = SCStream::new(filter, config, SomeErrorHandler {});

    let stream_output = SomeOutputWrapper { tx };
    stream.add_output(stream_output);
    stream.start_capture();

    let sample_buf = rx.recv().unwrap();

when trying to pass a type of stream output it doesn't the option to although i see in the code that there is apart from that CMSampleBuffer doesn't have all properties too

impl StreamOutput for SomeOutputWrapper {
    fn stream_output(&self, sample: CMSampleBuffer) {
        self.tx.send(sample).ok().unwrap()
    }
}

This is what sample_buf returns

CMSampleBuffer { presentation_timestamp: CMTime { value: 41655091430458, timescale: 1000000000, flags: 1, epoch: 0 }, pixel_buffer: Some(0x600002fde1b0) }

in the crate it's

use screencapturekit_sys::stream_output_handler::{CMSampleBuffer, UnsafeSCStreamOutput};

pub trait StreamOutput: Sync + Send + 'static {
    fn stream_output(&self, sample: CMSampleBuffer);
}

pub(crate) struct StreamOutputWrapper<T: StreamOutput>(T);

impl<T: StreamOutput> StreamOutputWrapper<T> {
    pub fn new(output: T) -> Self {
        Self(output)
    }
}

impl<TOutput: StreamOutput> UnsafeSCStreamOutput for StreamOutputWrapper<TOutput> {
    fn got_sample(&self, sample: CMSampleBuffer) {
        self.0.stream_output(sample);
    }
}
``` but in the repo it's something entirely different

Edit: I have tried cloning the repo and then adding the sckit to the deps 
```toml
screencapturekit = {version = "0.1.0", path="./screencapturekit-rs/screencapturekit"}

which gives me the latest methods but it crashes the thread

impl StreamOutput for SomeOutputWrapper {
    fn did_output_sample_buffer(
        &self,
        sample_buffer: screencapturekit::cm_sample_buffer::CMSampleBuffer,
        of_type: screencapturekit::sc_output_handler::SCStreamOutputType,
    ) {
        self.tx.send(sample_buffer).ok().unwrap()
    }
}

Crashed Thread:        0  main  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000368e91ff0390
Exception Codes:       0x0000000000000001, 0x0000368e91ff0390

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [59528]

Implement SCContentSharingPicker

See: https://developer.apple.com/documentation/screencapturekit/sccontentsharingpicker

Why not use bindgen for screencapturekit-sys?

I am looking for an audio capture crate and find this. Thank you for your effort! It seems that you write screencapturekit-sys by yourself, why not choose automatic generate tools like bindgen? And I find 0.1.0 version doesn't offer audio capture function but seems you updated it recently, do you have any plan to publish next version?

minimum_frame_interval not working as expected

Hey! Great work on the project it's great to use.

I am facing an issue with this stream configuration,
SCStreamConfiguration { shows_cursor: true, width: 100, height: 100, scales_to_fit: true, pixel_format: PixelFormat::ARGB8888, minimum_frame_interval: CMTime { value: 3, timescale: 1, flags: 1, epoch: 0, }, ..Default::default() };

According to this CMTime, I should be getting 1 frame every 3 seconds, but it doesn't seem to be respected as I am still getting 60fps.

Better error management (No panics)

Use https://github.com/dtolnay/thiserror and better error enums.

Can no longer import SCFrameStatus

Hi, I was looking at other repo's integrating this package and I noticed that they could import SCFrameStatus from sc_sys but I think that is no longer possible? Not sure which version omitted it.

Segmentation Fault while using 0.2.1 version

Previously, I was using a older version of the library and the screen capturing logic was working as expected. I've recently updated my code to use the latest version of screencapturekit-rs ( 0.2.1 ) and I've started seeing a Segmentation Fault.

Here's a sample code which reproduces the issue -

use std::{path::PathBuf, time, thread};

use core_graphics::display::{CGMainDisplayID, CGDisplay};
use screencapturekit::{sc_error_handler::StreamErrorHandler, sc_output_handler::{StreamOutput, SCStreamOutputType}, cm_sample_buffer::CMSampleBuffer, sc_content_filter::{InitParams, SCContentFilter}, sc_stream_configuration::SCStreamConfiguration, sc_stream::SCStream, sc_shareable_content::SCShareableContent};
use screencapturekit_sys::os_types::geometry::{CGRect, CGPoint, CGSize};

struct ErrorHandler;
impl StreamErrorHandler for ErrorHandler {
    fn on_error(&self) {
        println!("Error!");
    }
}

pub struct Capturer {
}

impl Capturer {
    pub fn new() -> Self {
        println!("Capturer initialized");
        Capturer {}
    }
}

impl StreamErrorHandler for Capturer {
    fn on_error(&self) {
        eprintln!("ERROR!");
    }
}

impl StreamOutput for Capturer {
    fn did_output_sample_buffer(&self, sample: CMSampleBuffer, of_type: SCStreamOutputType) {
        println!("New frame recvd");
    }
}
fn main() {
    println!("Starting");

    let content = SCShareableContent::current();
    let displays = content.displays;

    let display_id = unsafe { CGMainDisplayID() };
    let display = displays
        .iter()
        .find(|display| display.display_id == display_id)
        .unwrap_or_else(|| {
            panic!("Main display not found");
        });
    let display = display.to_owned();


    let mode = CGDisplay::new(display_id).display_mode().unwrap();
    let scale = (mode.pixel_width() / mode.width()) as u32;
    let width = display.width * scale;
    let height = display.height * scale;

    let params = InitParams::Display(display);
    let filter = SCContentFilter::new(params);

    let stream_config = SCStreamConfiguration {
        width,
        height,
        ..Default::default()
    };

    let mut stream = SCStream::new(filter, stream_config, ErrorHandler);
    stream.add_output(Capturer::new(), SCStreamOutputType::Screen);

    stream.start_capture();

    let ten_millis = time::Duration::from_millis(10000);

    thread::sleep(ten_millis);

    stream.stop_capture();

    println!("Ended");

}

This is the output I see

Maybe I'm using some part incorrectly but would be great if someone could look into this and help me out. Let me know if you need more help with the reproduction.

Better readme

Current readme is not very informative about how to use the library.

I want to add a more concise version that is easier to follow.

No TBDs, and include contributors and info about how to contribute.

Updating stream configuration to use a different pixel format has no effect

I've initialized the stream configuration to use BGRA as the pixel format –

But the output data I get is still in the YCbCr format.

The reason I think the data is in YCbCr format is -

• CMSampleBuffer.pixel_buffer.plane_count is 2 but the documentation says BGRA is "Packed little endian" and should not have planes
• Converting the buffer data from YCbCr to RGB and then saving it to an image works. ( This should not have generated a valid image if the buffer data was in BGRA as the conversion algorithm would have been incorrect )

Let me know if you need more context or if you want me to try out more things

SCStreamFrameInfo.status always return Idle?

In my environment, SCStreamFrameInfo.status always returns SCFrameStatus::Idle...

screencapturekit-rs/screencapturekit-sys/src/sc_stream_frame_info.rs

Line 36 in 96f6976

return SCFrameStatus::Idle;