cosyneco/mediapipe.net

Pure .NET bindings for Google's MediaPipe.

mediapipe.net's Issues

how to work opencvsharp together?

how to work opencvsharp together?
mat type is not support
videocapture read a pic from camera.
how to use it?

Synchronize with MediaPipeUnity 0.9.1

How to set parameters for calculators?

Hello!
I have create grpah from pbtxt file, and try to modify calculator parameters in this graph, to change thresholds for BlazePose. I use branch fix-side-packets for my experiments. I generate the missing files from Protobuf.

In proto files, append namespace:

option csharp_namespace = "Mediapipe.Net.Framework.Protobuf";

and next run:

protoc -I=. -I ..\..\..\ --csharp_out=.\cs_out\ .\tensors_to_detections_calculator.proto
protoc -I=. -I ..\..\..\ --csharp_out=.\cs_out\ .\thresholding_calculator.proto

i have attach generated files: CalculatorOptions.zip

GraphPath="mediapipe/modules/pose_landmark/pose_landmark_cpu.pbtxt"

CalculatorGraph GraphTmp = new CalculatorGraph(File.ReadAllText(GraphPath));
CalculatorGraphConfig configTmp = new CalculatorGraphConfig(GraphTmp.Config());

foreach (CalculatorGraphConfig.Types.Node n in configTmp.Node)
{
    if (n.Name.Equals("posedetectioncpu__TensorsToDetectionsCalculator"))
    {                   
        TensorsToDetectionsCalculatorOptions to = n.Options.GetExtension(TensorsToDetectionsCalculatorOptions.Extensions.Ext);
        to ??= new TensorsToDetectionsCalculatorOptions();
        to.MinScoreThresh = 0.99f;
        n.Options.SetExtension(TensorsToDetectionsCalculatorOptions.Extensions.Ext, to);
     }
     if (n.Name.Equals("poselandmarkbyroicpu__tensorstoposelandmarksandsegmentation__ThresholdingCalculator"))
    {
         ThresholdingCalculatorOptions to = n.Options.GetExtension(ThresholdingCalculatorOptions.Extensions.Ext);
         to ??= new ThresholdingCalculatorOptions();
         to.Threshold = 0.99;
         n.Options.SetExtension(ThresholdingCalculatorOptions.Extensions.Ext, to);
    }
}

//Now I'm doing some experiments with the graph:

// create new CalculatorGraph from modifyed config
Graph = new CalculatorGraph(configTmp); 

byte[] cb = configTmp.ToByteArray();
Console.WriteLine("Modified config length:" + cb.Length);  //Length 21759

//test merge graph from copy bytes
CalculatorGraphConfig c2 = new CalculatorGraphConfig();
c2.MergeFrom(cb);
byte[] cb2 = c2.ToByteArray(); 
Console.WriteLine("Merge copy, from modifyed config:" + cb2.Length); //Length 21759

//returned back graph
byte[] cb3 = Graph.Config().ToByteArray();
Console.WriteLine("Config returned back from Mediapipe.Runtime length:" + cb3.Length); //Length 21732

//Original graph that i read from pbtxt file
byte[] cb4 = GraphTmp.Config().ToByteArray();
Console.WriteLine("Original graph that we read from file:" + cb4.Length); //Length 21732

Output in console:

Modified config length:21759
Merge copy, from modifyed config:21759
Config returned back from Mediapipe.Runtime length:21732
Original graph that we read from file:21732

Those. the size of the graph has become smaller, in the version that went to Mediapipe.Runtime and returned back. We've lost extension options.

Another interesting point, is if I want to get values from an extension, I can only get it from the original graph, but not copies (even can't do it from byte array copy with same size). From any copy, this returns null.

TensorsToDetectionsCalculatorOptions? to1 = n.Options.GetExtension(TensorsToDetectionsCalculatorOptions.Extensions.Ext);

And in principle, these changes do not affect the result of the pipeline in any way. those. with a value of 0.99 I shouldn't have any detections at all. But they are exactly the same as before.
Thank you!

Setup NuGet packaging

If possible, we should work on NuGet publishing in the CI since this project is in a usable state for Encore. This would allow the client to start working on integrating everything.

`ResourceManager` delegates are not safe from garbage collection

When creating a child of the ResourceManager class, it will execute SafeNativeMethods.mp__SetCustomGlobalPathResolver__P(ResolvePath);, which is actually syntactic sugar for SafeNativeMethods.mp__SetCustomGlobalPathResolver__P(new PathResolver(ResolvePath));.
The same is true for SafeNativeMethods.mp__SetCustomGlobalResourceProvider__P(provideResource);, which is syntactic sugar for SafeNativeMethods.mp__SetCustomGlobalResourceProvider__P(new UnsafeResourceManager(provideResource));.

As C# doesn't care at all about the native side, this newly instantiated delegate may be garbage collected at any time during the program, and it will cause an error similar to this:

Process terminated. A callback was made on a garbage collected delegate of type 'Mediapipe.Net!Mediapipe.Net.Native.SafeNativeMethods+UnsafeResourceProvider::Invoke'.

We need to keep these delegates alive to avoid these problems.
This will be a good opportunity to refactor the ResourceManager class into a static solution, because the current singleton way really isn't the best.

Add examples

Now that the framework has been fully ported, it is time to add examples on how to make use of it.

A cool thing to do would be to have one that uses Moetion to get some solved tracking solution.
I can also work on one that makes use of osu!framework, as I've used it to make some tests with MediaPipe back then.

Timestamps shouldn't rely on current frame

This way of creating timestamps is actually wrong.
We should actually use the current time in microseconds, like we did in Akihabara.
https://github.com/vignetteapp/MediaPipe.NET/blob/73cd0429f8e1d75644ca5d63db8974a7a9b08928/Mediapipe.Net/Calculators/CpuCalculator.cs#L29

How to extract facial landmarks in xamarin forms app

Can you provide sample code or documentation to extract facial landmarks from image in a xamarin forms app.

BlazePoseCpuCalculator crashing

I've tried using the code from Mediapipe.Net.Examples.FaceMesh with BlazePoseCpuCalculator (after adding necessary medipipe / graphs and modules folder) on Windows 10, and I get the following exception

Where can I find a working example for this calculator?

Discussion on unmanaged memory leak

Based on what @Speykious and I dug, we narrowed down our memory leak to our managed code not calling the unmanaged deleters, leaving us with dangling pointers that has not been cleared.

Related Literature

@dbruning findings and memory profile shared to us

Memory Footprints of MediaPipe natively by @Speykious

Revisit Calculator Solutions

Since we've stabilized the API a bit, it's fair game to revisit the solutions again from the older versions to ease development.

Adding hair segmentation

Hi
Do you have the plan to add this feature to the library?
I wanted to add hair segmentation to the library.
how I can do that? and what I need to consider.
Thanks for the amazing work.

Let `ImageFrame` take pixel data as a `ReadOnlySpan<byte>` instead of a `byte*`

This will improve the API by making it possible to disallow unsafe blocks for high level examples.
While we want to use unsafe in place of any kind of marshaling as much as possible internally, externally we would want to hide it as much as possible (by default, that is).
Related constructor: https://github.com/vignetteapp/MediaPipe.NET/blob/73cd0429f8e1d75644ca5d63db8974a7a9b08928/Mediapipe.Net/Framework/Format/ImageFrame.cs#L34-L50

Are there plans to support ARM64? Windows 11 on ARM64

Are there plans to support ARM64? Windows 11 on ARM64

Replace all `Marshal` operations with `unsafe` ones

Marshalling in C# can be extremely slow. Thus, now that .NET 6 has come with NativeMemory, it is preferrable to use unsafe operations as much as we can.

One first step has been done: instead of using Unity's NativeArray for the ImageFrame class, we have used a raw byte* pointer and the tests worked flawlessly.

Nothing has been benchmarked so far, but we can infer from how marshalling works that using unsafe will benefit us in the long run.

Redo examples

Right now we have updated to 0.9.1 but our examples hasn't been updated. This will resolve #47.

Getting this error - System.IO.DirectoryNotFoundException: 'Could not find a part of the path '/mediapipe/graphs/face_mesh/face_mesh_desktop_live.pbtxt

System.IO.DirectoryNotFoundException: 'Could not find a part of the path '/mediapipe/graphs/face_mesh/face_mesh_desktop_live.pbtxt

I'm getting this error when I try to initialize the facemeshcpucalculator in a MAUI project.

Add Readme

Hi & thanks for your work on this project!
I'm trying to understand what this project actually does. I understand that it has evolved from Akihabara, and that you don't have a readme in place yet, but I'm looking at options to implement blazepose in my WPF application and this looks like one of them.

Once this project is a little further along, can you tell me what you expect the runtime dependencies to be? Does it need Python and GCC to run? Or just to build? Where does it run the machine learning models - tflite? Is Windows even a supported target?

My other option is to re-implement what https://github.com/tensorflow/tfjs-models is doing, in pose-detection/src/blazepose_tfjs/detector.ts - which I think is kind of a custom implementation of the "glue" parts of mediapipe, using tensorflow to actually run the models. But if I'm making that effort, I might be better to instead contribute to this project.

Thanks for any guidance.

ImageFrame SignalAbort tests failing due to incorrect ByteDepth

========== Starting test run ==========
NUnit Adapter 4.0.0.0: Test execution started
Running selected tests in D:\VgProjects\MediaPipe.NET\Mediapipe.Net.Tests\bin\Debug\net6.0\Mediapipe.Net.Tests.dll
   NUnit3TestExecutor discovered 18 of 18 NUnit test cases using Current Discovery mode, Non-Explicit run
The active test run was aborted. Reason: Test host process crashed : WARNING: Logging before InitGoogleLogging() is written to STDERR
F20230228 12:36:14.229180 24912 image_frame.cc:379] Check failed: 1 == ByteDepth() (1 vs. 2) 
*** Check failure stack trace: ***

========== Test run aborted: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in < 1 ms ==========

If anyone's willing to take this on, do submit a patch, but I'm currently stumped on this because I've been trying to debug the code for any potential things I missed in the native API.

NormalizedLandmarkList.Landmark is wrong on BlazePose

I have downloaded your latest master branch to test your BlazePose example for one of the projects I'm conducting. I got to run the console app successfully. then I have saved the "Landmark" from 200 frames in a file. The data in the first 5-10 frames are correct the rest of the data is being copied again and again. I managed to save the returning ImageFrame objects from Calculator.Send(..) as Jpg using ImageSharp - 200 images were saved - the images are changing (correctly masked my body) while the skeleton is not changing after 5-10 images.

I ran the same test for multiple camera positions, different movements, body poses and even clothes. each single time the same exact scenario mentioned.

Build Errors

Hi,

I am just trying to build a simple example project such us FaceMesh. I don't know if I have to do any configuration before build.

I have errors about packets. Should I install an specific version of a packet from nugget.

Any help about run a simple project should be helpful.

Thanks.

Unable to find an entry point

I have an error while creating ImageFrame for FaceMeshCpuCalculator:
Unable to find an entry point named 'mp_ImageFrame__ui_i_i_i_Pui8_PF' in DLL 'mediapipe_c'.

Here is part of the code:
var frame = camera.GetFrame();
converter = new FrameConverter(frame, PixelFormat.Rgba);
Frame? cFrame = converter.Convert(frame);
imgframe = new ImageFrame(ImageFormat.Srgba, cFrame.Width, cFrame.Height, cFrame.WidthStep, cFrame.RawData);//this row causes error

Here is libraries:
using Mediapipe.Net.Framework.Protobuf;
using Mediapipe.Net.Calculators;
using Mediapipe.Net.Framework.Format;
using SeeShark;
using SeeShark.Device;
using SeeShark.FFmpeg;

Pls give my piece of advice pls, i really need your help.

how to use local image from facemesh?

i want to us local image ,like this E:\aaaa\6_1\66666.jpg,
but Mediapipe.Net.Framework.Format.ImageFrame Unable to load local image ，
What should I do?

Thank you!

Incorporate Fuzzing into our Testing

As we are very prone of memory leaks than most projects, we must be extra careful and test for every occurence. For such a case we want to use a C#-based Fuzzer to test our managed code properly.

My proposed solution is use SharpFuzz to help us pinpoint issues like #18 and catch them in-CI.

Native pointer is not assigned when creating packet with default constructor

Thank you for porting this library from MediaPipeUnityPlugin. Wonderful work.

I noticed that there are mismatches in the default constructors of various Packet subclasses (e.g., DetectionVectorPacket, ImageFramePacket, etc.) between this project and MediaPipeUnityPlugin. In MediaPipeUnityPlugin, the default constructors call the base(true) base constructor, while the constructors in this project call base().

Using DetectionVectorPacket as example. The MediaPipeUnityPlugin project has the following.
https://github.com/homuler/MediaPipeUnityPlugin/blob/v0.9.1/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/DetectionVectorPacket.cs#L17

    public DetectionVectorPacket() : base(true) { }

And DetectionVectorPacket in this project has the following.
https://github.com/vignetteapp/MediaPipe.NET/blob/176415561c442cce8e92d1cde59b875e6a14995a/Mediapipe.Net/Framework/Packets/DetectionVectorPacket.cs#L15

Any particular reason on calling a different base constructor? If there is not any, can you please update default constructors of all Packet subclasses to call base(true) instead? Without base(true), native pointer to Packet is never assigned.

https://github.com/vignetteapp/MediaPipe.NET/blob/176415561c442cce8e92d1cde59b875e6a14995a/Mediapipe.Net/Framework/Packets/Packet.cs#L16-L26

Also, if it is not too much trouble, I would really appreciate if you can upload new NuGet packages with this change. Thank you so much.

The type or namespace name 'Solutions' does not exist in the namespace 'Mediapipe.Net'

Excuse me, how to solve this error, thank you

Properly separate CPU and GPU tests

Due to how the libraries are being used (one package reference for CPU and another for GPU, both incompatible with each other), the way we currently have to switch between CPU and GPU when performing test is by switching the corresponding package reference in Mediapipe.Net.Tests.csproj.
https://github.com/vignetteapp/MediaPipe.NET/blob/0eaa7242701ff16261934d9c8ccc95abe8cb5e57/Mediapipe.Net.Tests/Mediapipe.Net.Tests.csproj#L13-L14

This is not good for CI but also for the .csproj file as it's not good practice to constantly change the package references.
We have to find a better way to handle the different tests and packages.

Maybe by putting all GpuOnly tests into a different project called Mediapipe.Net.Tests.GPU?

should face_mesh_desktop_live.pbtxt be included in the repo?

Trying to run the samples, and the FaceMesh sample throws error:

Could not find a part of the path '<...>\Mediapipe.Net.Examples.FaceMesh\bin\Debug\net6.0\mediapipe\graphs\face_mesh\face_mesh_desktop_live.pbtxt'.

... maybe face_mesh_desktop_live.pbtxt and face_mesh_desktop_live_gpu.pbtxt just haven't been added to the repo yet?

Replace some methods by C# getters and setters

Now that MediaPipeUnityPlugin v0.8.2 for MediaPipe v0.8.9 has been ported one-to-one, we can start worrying about making changes to the framework so that it is nicer to use.

Some methods could be turned into C# properties for slightly better readability. Some examples are:
https://github.com/vignetteapp/MediaPipe.NET/blob/0eaa7242701ff16261934d9c8ccc95abe8cb5e57/Mediapipe.Net/Framework/Timestamp.cs#L42-L54
https://github.com/vignetteapp/MediaPipe.NET/blob/0eaa7242701ff16261934d9c8ccc95abe8cb5e57/Mediapipe.Net/Framework/CalculatorGraph.cs#L200-L215
https://github.com/vignetteapp/MediaPipe.NET/blob/0eaa7242701ff16261934d9c8ccc95abe8cb5e57/Mediapipe.Net/Framework/Format/ImageFrame.cs#L72-L114

About OpenCV's MAT converted to ImageFrame

Currently I am integrating this project into a desktop program and need to transfer opencv's mat data to ImageFrame, can this process be easily implemented?

use image file

how to use a png image as a source for pose estimation?

ImageFrame attempts to call a disposed method

The line: https://github.com/vignetteapp/MediaPipe.NET/blob/ce85bd59a4323e110596597797540fb2b30b56d1/Mediapipe.Net/Framework/Format/ImageFrame.cs#L47

is synactic sugar for:

new Deleter(releasePixelData),

Since this allocates a new object and not to mention the call is in line, this will be a gen 0 object that can be garbage collected at any moment and any attempts to call this method will cause a crash.

Support .NET 5.0

Why is .NET 5.0 not supported?

Alternate solution of Mediapipe.Net.Solutions

          Mediapipe.Net.Solutions no longer exists as of 0.9.1-mpu-0.9.1 as there are no proper maintainers for that API surface. Please consider using the raw MediaPipe API provided in the managed layer.

Originally posted by @sr229 in #49 (comment)

Could you elaborate more on the solution suggested (using the raw MediaPipe API provided in the managed layer)?

Epic: libmuxr to replace current runtime

Due to the myriad of issues we have encountered with our current wrapper, it's sufficient to say that we cannot trust the wrapper we derived from homuler due to testability, reproducibility of issues, and everything is just a black hole to us. Therefore, the next major task is re-architecting the wrapper to a new implementation.

Introduction to MUXR: The MediaPipe Universal eXtension Runtime

libmuxr, or simply MUXR, is our answer to a lot of issues we've encountered during the creation of MediaPipe.NET and porting MediaPipeUnity as Akihabara. MUXR aims to do the following:

• MUXR should handle every pointer ownership by MediaPipe. Any implementing API on top of it should not be able to touch the MediaPipe pointers as much as possible.
• Produce a Facade approach to the API, which is, make a C-compatible and convenient library as much as possible: which opens MUXR to be integrated by other languages if we wish to which only supports C ABIs.
• Invoking MUXR should be as easy as instantiating a new MediaPipe context in the Browser.

Of course a lot of the APIs we use like custom resources should still be supported, but we will have to re-architect everything including the wrapper to accomodate this new architecture.

`SignalAbort` tests make MediaPipe.NET completely crash on Windows

The issue seems to be the same on the MediaPipeUnityPlugin side. As Homuler pointed out back then:

We didn't investigate the C bindings side enough to be able to fix this right away, but it does seem like it is related to Windows not catching an abort signal fast enough for MediaPipe not to crash on the native side.

Additional help on this would be appreciated!

Update face capture to 52 blendshapes

Feature Description

The following commit provides the latest mediapipe 52 blendshapes
google/mediapipe@ba10ae8

The geometry and coefficients are listed here

With this new addition, this provides facial mocap for avatars that use the ARKit 52 blendshapes.

Current Behaviour/State

Currently, Users who are using industry standards e.g. Character Creator 3 limit themselves only facial mocap apps from Apple App Store

Questions about graphPath path settings

I encountered a problem MediaPipe.Net using the Windows App SDK application framework, the program will find the mediapipe directory from the path of System C:WindowsSystem32 by default, so I want to set it in the form of a custom directory, I don't know if this function can be opened.