Giter Site home page Giter Site logo

pixpark / gpupixel Goto Github PK

View Code? Open in Web Editor NEW
982.0 11.0 125.0 80.55 MB

Real-time image and video processing library similar to GPUImage, with built-in beauty filters, achieving commercial-grade beauty effects. Written in C++11 and based on OpenGL/ES.

License: MIT License

Java 9.25% Objective-C 3.20% Objective-C++ 4.39% C 0.40% CMake 12.52% C++ 70.24%
beautyface filter gpu android ios bigeye thinface gpuimage image opengl

gpupixel's Introduction

English | 简体中文

GPUPixel Stars GPUPixel Release GPUPixel Stars GPUPixel Stars


Welcome to join us to make GPUPixel better by participating discussions, opening issues, submitting PRs.

Introduction

⛰️ GPUPixel is a real-time, high-performance image and video filter library, extremely easy to compile and integrate with small library size.

🔑 GPUPixel is written in C++11 and is based on OpenGL/ES, incorporating a built-in beauty face filter that achieves commercial-grade beauty effects.

🔌 GPUPixel supports platforms including iOS, Android, Mac, Win and Linux. It can theoretically be ported to any platform that supports OpenGL/ES.

Effects Preview

👉 Video: YouTube | BiliBili

Origin Smooth White ThinFace
BigEye Lipstick Blusher ON-OFF

Architecture

Features

This table compares the features supported by GPUPixel and GPUImage and Android-GPUImage:

✅: Supported | ❌: Not supported | ✏️: Planning

GPUPixel GPUImage Android-GPUImage
📷 Filters:
Skin Smoothing Filter
Skin Whitening Filter
Face Slimming Filter
Big Eyes Filter
Lipstick Filter
Blush Filter
More Build in Filter
🎬 Input Formats:
YUV420P(I420)
RGBA
JPEG
PNG
NV21(for Android) ✏️
🎥 Output Formats:
RGBA
YUV420P(I420) ✏️
💻 Platform:
iOS
Mac
Android
Win
Linux

Performance

iPhone

- iPhone 6P iPhone 8 iPhone X iPhone 11 iPhone 14 pro
CPU 5% 5% 3% 3% 3%
Time Taken 10ms 4ms 3ms 3ms 3ms

Android

- Xiaomi 10 Huawei Mate30 Vivo SAMSUNG Google Pixel
CPU 3% 5% - - -
Time Taken 6ms 5ms - - -

Lib Size

iOS(.framework) MacOS(.framework) Android(.aar)
Size 2.4 MB 2.6 MB 2.1 MB

Before You Start

⭐️ Star us on GitHub, and be instantly notified for new releases!

Getting Started

How to Build

Compile using CMake frome v1.1.0. Please search for instructions on how to install and configure CMake.

The generated libraries and app will be located in the "output" directory of the project's root folder.

iOS

# Generate project
## for iOS arm64
cmake -G Xcode -B build -S src -DCMAKE_TOOLCHAIN_FILE=toolchain/ios.toolchain.cmake -DPLATFORM=OS64
 
# Build
cmake --build build --config Debug #or Release

Mac

# Generate project
## for Mac Apple Silicon
cmake -G Xcode -B build -S src -DCMAKE_TOOLCHAIN_FILE=toolchain/ios.toolchain.cmake -DPLATFORM=MAC_ARM64
## for Mac Intel
cmake -G Xcode -B build -S src -DCMAKE_TOOLCHAIN_FILE=toolchain/ios.toolchain.cmake -DPLATFORM=MAC

# Build
cmake --build build --config Debug #or Release

Android

Open the directory src/android/java in Android Studio.

Windows

You need install and config Cmake and MinGW64 by yourself.

# Generate project
cmake -G "MinGW Makefiles" -B build -S src

# Build
cmake --build build -DCMAKE_BUILD_TYPE=Debug #or Release

Linux (Test on ubuntu)

# install cmake 
sudo apt-get install cmake pkg-config
# install dependent lib
sudo apt-get install mesa-utils libglu1-mesa-dev freeglut3-dev mesa-common-dev libglfw3-dev

# Generate project
cmake -B build -S src

# Build
cmake --build build -DCMAKE_BUILD_TYPE=Debug #or Release 

App demo

iOS and Mac

Refer to examples/ios and examples/mac

Android

Refer to src/android/java

Window and Linux

Refer to examples/desktop . The compilation method is the same as compiling the library.

# Generate project
cmake -G "MinGW Makefiles" -B examples/build -S examples

# Build
cmake --build build -DCMAKE_BUILD_TYPE=Debug #or Release 

App Usage

A S D F G H - Increase smooth, white, thin face, big eye, lipstick, blusher level

Z X C V B N - Decrease smooth, white, thin face, big eye, lipstick, blusher level

How to Use

Declear filters

// video data input
std::shared_ptr<SourceRawDataInput> source_raw_input_;
// beauty filter
std::shared_ptr<BeautyFaceFilter> beauty_face_filter_;
// video data output 
std::shared_ptr<TargetRawDataOutput> target_raw_output_;

Create and link filters

 gpupixel::GPUPixelContext::getInstance()->runSync([&] {
    // Create filter
    source_raw_input_ = SourceRawDataInput::create();
    target_raw_output_ = TargetRawDataOutput::create();
    // Face Beauty Filter
    beauty_face_filter_ = BeautyFaceFilter::create();
    
    // Add filter
    source_raw_input_->addTarget(beauty_face_filter_)
                     ->addTarget(target_raw_output_);
 }

Input YUV420P or RGBA

// ...
// YUVI420
 source_raw_input_->uploadBytes(width,
                                height, 
                                bufferY,
                                strideY, 
                                bufferU, 
                                strideU,
                                bufferV, 
                                strideV);
// ...
// bytes: RGBA data
 source_raw_input_->uploadBytes(bytes,
                                width, 
                                height, 
                                stride);

Output Data Callback

// I420 callback
target_raw_output_->setI420Callbck([=](const uint8_t *data, 
                                        int width, 
                                        int height, 
                                        int64_t ts) {
    size_t y_size = width * height;
    const uint8_t *uData = data + y_size;
    const uint8_t *vData = data + y_size + y_size / 4;
    // Do something you want
});

// RGBA callback->
target_raw_output_->setPixelsCallbck([=](const uint8_t *data, 
                                        int width, 
                                        int height, 
                                        int64_t ts) {
    size_t rgba_size = width * height*4;
    // Do something you want
});

// Output data callbck

Contributing

Welcome to join us to make GPUPixel better by participating discussions, opening issues, submitting PRs.

At the same time, please consider supporting GPUPixel by sharing it on social media and at events and conferences.

Contributors

Acknowledgement

Reference Project

  1. GPUImage
  2. CainCamera
  3. AwemeLike
  4. VNN

License

This repository is available under the MIT License.

gpupixel's People

Contributors

cijiugechu avatar gezhaoyou avatar liamkeh avatar xiaoshanlin000 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gpupixel's Issues

Android version run failed in main branch

I try to run it in Android Studio with emulator, but it runs failed. The root cause is: in 'build.gradle', it defines targetSdk to '31', but in the code (GPUPixelSourceCamera.java) is still using the 'Camera' APIs which have been deprecated. Or anything am I doing wrong?

哪里有windows 的 vnn_core.dll 下载?

Describe the bug
A clear and concise description of the bug.

How To Reproduce the bug
Steps to reproduce the behavior, how frequent can you experience the bug:
1.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Running Information(please complete the following information):

  • GPUPixel Version using : [e.g. 1.2.1]
  • Operating System: [e.g. Ubuntu 20.04, Windows 11, iOS 15.2, Android 13]
  • Device Info: [e.g. Huawei Mate40, Xiaomi 10]

Additional context
Add any other context about the problem here.

[BUG] Error encountered during compilation for MacOS

An error occurred while compiling under MacOS

path: example/build

error:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.4.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSObject.h:9:1: error: 
      expected unqualified-id
@class NSInvocation, NSMethodSignature, NSCoder, NSString, NSEnumerator;
image

[Linux] Linux demo can't work on main branch

2 issues:

  1. Linux demo can't be linked as VNN lib directory is not added in link path. To fix it, please refer attached picture.
  2. The demo can run, but the FB doesn't take effect, the output image is not beatified. Please refer the attached picture.
    MicrosoftTeams-image (5)
    MicrosoftTeams-image (4)

[Android] How to use GPUPixelSourceImage?

Platform/Device: Android 14 - Xiaomi 11T 5G

GPUPixel Version: 1.0.6

I use GpuPixel in WebRTC, each VideoFrame processes GpuPixel to apply beauty filter. Below is my code:

fun uploadSourceToGPUPixel(originalBitmap: Bitmap) {
        try {
            if (originalBitmap.width > 0) {
                processingFrame = true

                runBlocking {
                    val sourceImage = GPUPixelSourceImage(originalBitmap)

                    sourceImage.captureAProcessedFrameData(beautyFaceFilter) { result ->
                        Log.d(tag, "---------FRAME PROCESSED---------")
                        callback?.onResult(result ?: originalBitmap)
                        processingFrame = false
                    }
                }
            }
        } catch (error: Exception) {
            Log.e(tag, error.toString())
        }
    }

But my app crash with error logs:

2024-02-15 16:45:43.023  1625-13322 AISPQSelector           camerahalserver                      E  [DpEngineAALReConfigFrame] aispq_en: 0
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  Build fingerprint: 'Xiaomi/amber_global/amber:14/UP1A.231005.007/V816.0.3.0.UKWMIXM:user/release-keys'
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  Revision: '0'
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  ABI: 'arm64'
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  Timestamp: 2024-02-15 16:45:42.917734998+0700
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  Process uptime: 4351s
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  Cmdline: com.waterbus.wanted
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  pid: 9825, tid: 13815, name: Thread-24  >>> com.waterbus.wanted <<<
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  uid: 10729
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xffffffff00000008
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A      x0  b4000079977a1180  x1  000000799502c748  x2  0000000000000000  x3  0000000000000000
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A      x4  0000000000000000  x5  0000000000000000  x6  0000000000000043  x7  0000000000000000
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A      x8  ffffffff00000000  x9  0000000000000000  x10 ffffffffffffffff  x11 0000000000000000
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A      x12 0000000000000000  x13 000000799502b2d0  x14 0000007af251ea01  x15 00000000ebad6a89
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A      x16 00000079745f6eb0  x17 0000007974518acc  x18 0000007947fee000  x19 b4000079ba75b700
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A      x20 b400007a43c182a0  x21 00000000000001e0  x22 0000000000000168  x23 b4000079ba75b700
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A      x24 000000799502d000  x25 000000799502c8c0  x26 000000799502c8e4  x27 000000799502c8c0
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A      x28 000000799502c7b0  x29 000000799502c730
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A      lr  0000007974542cb4  sp  000000799502c710  pc  0000007974522ec4  pst 0000000040001000
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  14 total frames
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A  backtrace:
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #00 pc 0000000000071ec4  /data/app/~~4K4ZWsbcPj-USsnBCD1-cQ==/com.waterbus.wanted-vsBh_6tS-2BV5IhoR44pWA==/base.apk!libgpupixel.so (offset 0x1c11000) (std::__ndk1::__shared_ptr_pointer<GPUPixel::Filter*, std::__ndk1::default_delete<GPUPixel::Filter>, std::__ndk1::allocator<GPUPixel::Filter> >::__on_zero_shared()+12) (BuildId: 59441ca5dcd7bf4287e6ee971a172bd7897bc944)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #01 pc 0000000000091cb0  /data/app/~~4K4ZWsbcPj-USsnBCD1-cQ==/com.waterbus.wanted-vsBh_6tS-2BV5IhoR44pWA==/base.apk!libgpupixel.so (offset 0x1c11000) (GPUPixel::Source::captureAProcessedFrameData(std::__ndk1::shared_ptr<GPUPixel::Filter>, int, int)+268) (BuildId: 59441ca5dcd7bf4287e6ee971a172bd7897bc944)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #02 pc 00000000000713cc  /data/app/~~4K4ZWsbcPj-USsnBCD1-cQ==/com.waterbus.wanted-vsBh_6tS-2BV5IhoR44pWA==/base.apk!libgpupixel.so (offset 0x1c11000) (Java_com_pixpark_gpupixel_GPUPixel_nativeSourceCaptureAProcessedFrameData+116) (BuildId: 59441ca5dcd7bf4287e6ee971a172bd7897bc944)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #03 pc 0000000000355630  /apex/com.android.art/lib64/libart.so (art_quick_generic_jni_trampoline+144) (BuildId: 02bec5940be704b863f6514fc7d81c41)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #04 pc 00000000005bb68c  /apex/com.android.art/lib64/libart.so (nterp_helper+3852) (BuildId: 02bec5940be704b863f6514fc7d81c41)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #05 pc 000000000133ae8a  /data/app/~~4K4ZWsbcPj-USsnBCD1-cQ==/com.waterbus.wanted-vsBh_6tS-2BV5IhoR44pWA==/oat/arm64/base.vdex (com.pixpark.gpupixel.GPUPixelSource$5.run+130)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #06 pc 00000000005bc4f4  /apex/com.android.art/lib64/libart.so (nterp_helper+7540) (BuildId: 02bec5940be704b863f6514fc7d81c41)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #07 pc 000000000133a4fc  /data/app/~~4K4ZWsbcPj-USsnBCD1-cQ==/com.waterbus.wanted-vsBh_6tS-2BV5IhoR44pWA==/oat/arm64/base.vdex (com.pixpark.gpupixel.GPUPixel$3.run+4)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #08 pc 0000000000505818  /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (java.lang.Thread.run+72)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #09 pc 000000000033eba4  /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612) (BuildId: 02bec5940be704b863f6514fc7d81c41)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #10 pc 000000000023a9ac  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+144) (BuildId: 02bec5940be704b863f6514fc7d81c41)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #11 pc 000000000053b96c  /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1600) (BuildId: 02bec5940be704b863f6514fc7d81c41)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #12 pc 0000000000101d5c  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+204) (BuildId: 84a42637b3a421b801818f5793418fca)
2024-02-15 16:45:43.057 13818-13818 DEBUG                   pid-13818                            A        #13 pc 0000000000095bc0  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 84a42637b3a421b801818f5793418fca)

瘦脸算法是否支持实时视频

想请教一下作者,瘦脸算法是否可以支持实时视频使用,我尝试使用液波或本地平移算法对视频都没有很好的效果,作者有使用过类似的实时瘦脸算法吗,非常感谢

[Help] Integrate GPUPixel into WebRTC applications

Hi @LiamKeh ,

First of all, GPUPixel is very interesting, I'm trying to integrate GPUPixel into an Open Source WebRTC Video Conferencing application on MacOS, iOS and Android. I'm starting with MacOS first, and a few problems occurred. I have created a RTCBeautyFilter class, it looks like this:

#import "RTCBeautyFilter.h"
#import "gpupixel/gpupixel.h"

using namespace gpupixel;

@interface RTCBeautyFilter () {
    std::shared_ptr<SourceRawDataInput> gpuPixelRawInput;
    GPUPixelView *gpuPixelView;
    std::shared_ptr<BeautyFaceFilter> beauty_face_filter_;
    std::shared_ptr<TargetRawDataOutput> targetRawOutput_;
    std::shared_ptr<FaceReshapeFilter> face_reshape_filter_;
    std::shared_ptr<gpupixel::FaceMakeupFilter> lipstick_filter_;
    std::shared_ptr<gpupixel::FaceMakeupFilter> blusher_filter_;
}

@end

@implementation RTCBeautyFilter

- (instancetype)init {
    self = [super init];
    if (self) {
        [self setup];
    }
    return self;
}

- (void)setup {
    // Init video filter
    [self initVideoFilter];
}

- (void)initVideoFilter {
    gpuPixelRawInput = SourceRawDataInput::create();
 
    // Create filters
    lipstick_filter_ = LipstickFilter::create();
    blusher_filter_ = BlusherFilter::create();
    face_reshape_filter_ = FaceReshapeFilter::create();
    
    gpuPixelRawInput->RegLandmarkCallback([=](std::vector<float> landmarks) {
       lipstick_filter_->SetFaceLandmarks(landmarks);
       blusher_filter_->SetFaceLandmarks(landmarks);
       face_reshape_filter_->SetFaceLandmarks(landmarks);
     });
 
    // Create filter
    targetRawOutput_ = TargetRawDataOutput::create();
    beauty_face_filter_ = BeautyFaceFilter::create();
  
    gpuPixelRawInput->addTarget(lipstick_filter_)
                       ->addTarget(blusher_filter_)
                       ->addTarget(face_reshape_filter_)
                       ->addTarget(beauty_face_filter_)
                       ->addTarget(gpuPixelView);
  
    [gpuPixelView setFillMode:(gpupixel::TargetView::PreserveAspectRatioAndFill)];
}

#pragma mark - Property assignment

- (void)setBeautyValue:(CGFloat)value {
    _beautyValue = value;
    beauty_face_filter_->setBlurAlpha(value/10);
}

- (void)setWhithValue:(CGFloat)value {
    _whithValue = value;
    beauty_face_filter_->setWhite(value/20);
}

- (void)setThinFaceValue:(CGFloat)value {
    _thinFaceValue = value;
    face_reshape_filter_->setFaceSlimLevel(value/100);
}

- (void)setEyeValue:(CGFloat)value {
    _eyeValue = value;
    face_reshape_filter_->setEyeZoomLevel(value/50);
}

- (void)setLipstickValue:(CGFloat)value {
    _lipstickValue = value;
    lipstick_filter_->setBlendLevel(value/10);
}

- (void)setBlusherValue:(CGFloat)value {
    _blusherValue = value;
    blusher_filter_->setBlendLevel(value/10);
}

- (void)processVideoFrame:(CVImageBufferRef)videoFrame {
    // Process video frame here
    CVPixelBufferLockBaseAddress(videoFrame, 0);
    auto width = CVPixelBufferGetWidth(videoFrame);
    auto height = CVPixelBufferGetHeight(videoFrame);
    auto stride = CVPixelBufferGetBytesPerRow(videoFrame)/4;
    auto pixels = (const uint8_t *)CVPixelBufferGetBaseAddress(videoFrame);
    gpuPixelRawInput->uploadBytes(pixels, width, height, stride);
    CVPixelBufferUnlockBaseAddress(videoFrame, 0);
}

@end

and when I execute, the error appears:

ld: warning: Could not find or use auto-linked framework 'CoreAudioTypes': framework 'CoreAudioTypes' not found
ld: Undefined symbols:
  gpupixel::BlusherFilter::create(), referenced from:
      -[RTCBeautyFilter initVideoFilter] in RTCBeautyFilter.o
  gpupixel::LipstickFilter::create(), referenced from:
      -[RTCBeautyFilter initVideoFilter] in RTCBeautyFilter.o
  gpupixel::BeautyFaceFilter::setBlurAlpha(float), referenced from:
      -[RTCBeautyFilter setBeautyValue:] in RTCBeautyFilter.o
  gpupixel::BeautyFaceFilter::create(), referenced from:
      -[RTCBeautyFilter initVideoFilter] in RTCBeautyFilter.o
  gpupixel::BeautyFaceFilter::setWhite(float), referenced from:
      -[RTCBeautyFilter setWhithValue:] in RTCBeautyFilter.o
  gpupixel::FaceMakeupFilter::SetFaceLandmarks(std::__1::vector<float, std::__1::allocator<float>>), referenced from:
      -[RTCBeautyFilter initVideoFilter]::$_0::operator()(std::__1::vector<float, std::__1::allocator<float>>) const in RTCBeautyFilter.o
      -[RTCBeautyFilter initVideoFilter]::$_0::operator()(std::__1::vector<float, std::__1::allocator<float>>) const in RTCBeautyFilter.o
  gpupixel::FaceReshapeFilter::setEyeZoomLevel(float), referenced from:
      -[RTCBeautyFilter setEyeValue:] in RTCBeautyFilter.o
  gpupixel::FaceReshapeFilter::SetFaceLandmarks(std::__1::vector<float, std::__1::allocator<float>>), referenced from:
      -[RTCBeautyFilter initVideoFilter]::$_0::operator()(std::__1::vector<float, std::__1::allocator<float>>) const in RTCBeautyFilter.o
  gpupixel::FaceReshapeFilter::setFaceSlimLevel(float), referenced from:
      -[RTCBeautyFilter setThinFaceValue:] in RTCBeautyFilter.o
  gpupixel::FaceReshapeFilter::create(), referenced from:
      -[RTCBeautyFilter initVideoFilter] in RTCBeautyFilter.o
  gpupixel::SourceRawDataInput::uploadBytes(unsigned char const*, int, int, int, long long), referenced from:
      -[RTCBeautyFilter processVideoFrame:] in RTCBeautyFilter.o
  gpupixel::SourceRawDataInput::create(), referenced from:
      -[RTCBeautyFilter initVideoFilter] in RTCBeautyFilter.o
  gpupixel::TargetRawDataOutput::create(), referenced from:
      -[RTCBeautyFilter initVideoFilter] in RTCBeautyFilter.o
  gpupixel::Source::RegLandmarkCallback(std::__1::function<void (std::__1::vector<float, std::__1::allocator<float>>)>), referenced from:
      -[RTCBeautyFilter initVideoFilter] in RTCBeautyFilter.o

Is it correct to integrate GPUPixel into a WebRTC application? and how to fix the above error

单独使用图片进行滤镜处理

作者你好,我想咨询下如果我的需求是只使用图片进行这些滤镜的处理而不用取景框,并且能保存处理后的图片到本地,这个功能在现有的demo里可以实现吗
我有发现有一个GPUPixelSourceImage是可以直接传入图片的bitmap的,但是GPUPixelView这个类中只添加了surfaceview,是否我需要参照这个添加一个类只有一个imageview的子布局呢

[BUG]如何获取拍照的截图呢

iOS 使用 captureAProcessedFrameData这个函数返回的是个什么结果呢?如何转换成图片
安卓调用这个api,目前会发生崩溃

Android版本运行崩溃

pid: 5217, tid: 5250, name: GLThread 325 >>> com.pixpark.gpupixel <<<
2023-12-07 14:12:31.612 5334-5334 DEBUG pid-5334 A signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x108
2023-12-07 14:12:31.612 5334-5334 DEBUG pid-5334 A x0 0000000000000000 x1 0000000000000006 x2 0000000000000000 x3 0000000000000006
2023-12-07 14:12:31.612 5334-5334 DEBUG pid-5334 A x4 0000007554ffde48 x5 000000754146052a x6 0000000000000000 x7 0000000000000000
2023-12-07 14:12:31.612 5334-5334 DEBUG pid-5334 A x8 0000000000000000 x9 0000000000000006 x10 0000000000430000 x11 0000007554fff4e8
2023-12-07 14:12:31.612 5334-5334 DEBUG pid-5334 A x12 00000000000000c4 x13 0000000000000012 x14 000000756083a250 x15 dfcb56e8fb5eb3f5
2023-12-07 14:12:31.612 5334-5334 DEBUG pid-5334 A x16 0000007540f69638 x17 0000007540eb81b4 x18 000000006f56dc08 x19 000000755c29f000
2023-12-07 14:12:31.612 5334-5334 DEBUG pid-5334 A x20 000000755bca9e50 x21 000000755c29f000 x22 0000007554ffdc4c x23 000000754146052a
2023-12-07 14:12:31.612 5334-5334 DEBUG pid-5334 A x24 000000000000000c x25 17e56458624f2ffe x26 000000755c29f098 x27 17e56458624f2ffe
2023-12-07 14:12:31.612 5334-5334 DEBUG pid-5334 A x28 0000000000000003 x29 0000007554ffd980 x30 0000007540e2b1d4
2023-12-07 14:12:31.612 5334-5334 DEBUG pid-5334 A sp 0000007554ffd950 pc 0000007540eb81c8 pstate 0000000020000000
2023-12-07 14:12:31.614 5334-5334 DEBUG pid-5334 A
backtrace:
2023-12-07 14:12:31.614 5334-5334 DEBUG pid-5334 A #00 pc 00000000001311c8 /data/app/com.pixpark.gpupixel-1/lib/arm64/libgpupixel.so (_ZN8GPUPixel18SourceRawDataInput11setRotationENS_12RotationModeE+20)
2023-12-07 14:12:31.614 5334-5334 DEBUG pid-5334 A #1 pc 00000000000a41d0 /data/app/com.pixpark.gpupixel-1/lib/arm64/libgpupixel.so (Java_com_pixpark_gpupixel_GPUPixel_nativeSourceRawInputSetRotation+40)
2023-12-07 14:12:31.614 5334-5334 DEBUG pid-5334 A #2 pc 000000000069cccc /data/app/com.pixpark.gpupixel-1/oat/arm64/base.odex (offset 0x603000)

Android : App crashed on start

I tried to run on android device ( Pixel 2 ). After granting camera access, the application crashed, displaying the following errors.

Screenshot 2024-02-01 at 4 57 18 in the afternoon

[BUG]

在android studio上起来了,项目也能跑起来,但是起来后就闪退,报错
mac电脑,华为真机测试机,报错如下,请指导一下
FATAL EXCEPTION: main
Process: com.pixpark.gpupixeldemo, PID: 27028
java.lang.RuntimeException: Unable to start activity ComponentInfo{com.pixpark.gpupixeldemo/com.pixpark.GPUPixelApp.MainActivity}: android.view.InflateException: Binary XML file line #14 in com.pixpark.gpupixeldemo:layout/activity_main: Binary XML file line #14 in com.pixpark.gpupixeldemo:layout/activity_main: Error inflating class com.pixpark.gpupixel.GPUPixelView
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:4765)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:4983)
at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:123)
at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:149)
at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:103)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:3059)
at android.os.Handler.dispatchMessage(Handler.java:117)
at android.os.Looper.loopOnce(Looper.java:205)
at android.os.Looper.loop(Looper.java:293)
at android.app.ActivityThread.loopProcess(ActivityThread.java:9934)
at android.app.ActivityThread.main(ActivityThread.java:9923)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:586)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1240)
Caused by: android.view.InflateException: Binary XML file line #14 in com.pixpark.gpupixeldemo:layout/activity_main: Binary XML file line #14 in com.pixpark.gpupixeldemo:layout/activity_main: Error inflating class com.pixpark.gpupixel.GPUPixelView
Caused by: android.view.InflateException: Binary XML file line #14 in com.pixpark.gpupixeldemo:layout/activity_main: Error inflating class com.pixpark.gpupixel.GPUPixelView
Caused by: java.lang.reflect.InvocationTargetException
at java.lang.reflect.Constructor.newInstance0(Native Method)
at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
at android.view.LayoutInflater.createView(LayoutInflater.java:868)
at android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:1020)
at android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:975)
at android.view.LayoutInflater.rInflate(LayoutInflater.java:1137)
at android.view.LayoutInflater.rInflateChildren(LayoutInflater.java:1098)
at android.view.LayoutInflater.inflate(LayoutInflater.java:696)
at android.view.LayoutInflater.inflate(LayoutInflater.java:540)
at com.android.internal.policy.HwPhoneLayoutInflater.inflate(HwPhoneLayoutInflater.java:138)
at com.pixpark.gpupixeldemo.databinding.ActivityMainBinding.inflate(ActivityMainBinding.java:90)
at com.pixpark.gpupixeldemo.databinding.ActivityMainBinding.inflate(ActivityMainBinding.java:84)
at com.pixpark.GPUPixelApp.MainActivity.onCreate(MainActivity.java:48)
at android.app.Activity.performCreate(Activity.java:8592)
at android.app.Activity.performCreate(Activity.java:8565)
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1344)
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:4733)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:4983)
at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:123)
at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:149)
at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:103)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:3059)
at android.os.Handler.dispatchMessage(Handler.java:117)
at android.os.Looper.loopOnce(Looper.java:205)
at android.os.Looper.loop(Looper.java:293)
at android.app.ActivityThread.loopProcess(ActivityThread.java:9934)
at android.app.ActivityThread.main(ActivityThread.java:9923)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:586)
2024-04-30 17:33:15.191 27028-27028 AndroidRuntime com.pixpark.gpupixeldemo E at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1240)
Caused by: java.lang.UnsatisfiedLinkError: dlopen failed: "/data/app/~~IFugzIY0v7TP8tkOdOCMiw==/com.pixpark.gpupixeldemo-0EPtT2Ej7lL5RGIQylmt0w==/lib/arm64/libgpupixel.so" is 32-bit instead of 64-bit
at java.lang.Runtime.loadLibrary0(Runtime.java:1077)
at java.lang.Runtime.loadLibrary0(Runtime.java:998)
at java.lang.System.loadLibrary(System.java:1657)
at com.pixpark.gpupixel.GPUPixel.(GPUPixel.java:160)
at com.pixpark.gpupixel.GPUPixel.getInstance(GPUPixel.java:48)
at com.pixpark.gpupixel.GPUPixelView.init(GPUPixelView.java:41)
at com.pixpark.gpupixel.GPUPixelView.(GPUPixelView.java:36)
... 30 more

安卓SDK集成问题

尝试把gpupixel接入一个安卓项目。首先使用example中的GPUPixelSourceCamera类可以正常工作。想拿到输出图像的数据做进一步处理。之前在windows平台集成sdk使用sourceImage输入处理画面并用captureAProcessedFrameData()方法也可以正常工作。在安卓中调用几帧后崩溃,附调用代码和堆栈图。请问是否可以指点一下

python api 的请求

在影视行业,有大量使用python 和 C++ 联合编程的案例来弥补python 在图形图像处理的性能缺陷。这个项目能否在未来有python api 接口,方便我们这些python 开发来将这个模块集成到特效软件中?

Question about FaceMakeupFilter landmarks

repo 的实时视频和图像的能力令人印象深刻,在研究您的项目时,我对其中的 FaceMakeupFilter 功能产生了一些疑问,希望能得到您的解答。

在 DEMO 中有实现两个 Filter,分别是blusher_filter_lipstick_filter_,它们是如何与landmarks协同去定位面部位置的呢?这里有几点没有理解:

  • 从代码看它们使用的是同一个类FaceMakeupFilter,唯一的差别是setTextureBounds的位置不同,是不是意味着 filters 并不知道什么叫嘴什么叫腮,它仅仅是根据TextureBounds去做形变?
  • 这两个滤镜的TextureBounds FrameBounds{502.5, 710, 262.5, 167.5}FrameBounds{395, 520, 489, 209}是如何得到的呢?如果我想要去实现一个新的滤镜,该如何确定这些值?
  • 我试了一下口红滤镜,它能够准确避免牙齿被上色,这是似乎是通过faceIndexs对嘴部的三角形进行着色实现的?但是mouth 是一张图片,并不是一个颜色,它是如何形变去适配嘴部区域的?尤其是考虑到前面的第一点:FaceMakeupFilter应该并不知道它的实例的想要美化的是嘴还是腮?

I'm thoroughly impressed with your real-time video and image processing library on GitHub. While exploring your project, I have some questions about the FaceMakeupFilter feature and would appreciate your insights.

In the DEMO, two filters are implemented: blusher_filter_ and lipstick_filter_. I'm intrigued by how they work in conjunction with facial landmarks. Here are my questions:

  • Both filters seem to use the same FaceMakeupFilter class, with the main difference being the placement of setTextureBounds. Does this mean that the filters don't directly recognize specific facial areas like the mouth or cheeks, and instead deform based on TextureBounds?
  • How are the TextureBounds for blusher_filter_ and lipstick_filter_ determined? For instance, how were values like FrameBounds{502.5, 710, 262.5, 167.5} and FrameBounds{395, 520, 489, 209} arrived at? If I want to implement a new filter, how should I determine the values for these TextureBounds?
  • I tried the lipstick filter and noticed its precise avoidance of coloring the teeth, apparently achieved by coloring the mouth's triangles using faceIndexs. However, if the mouth is represented as an image rather than a single color, how does it deform to fit the mouth area? This is particularly intriguing considering FaceMakeupFilter might not differentiate whether it's being used for the mouth or cheeks.

[Feature Request]是否支持在arm架构下的linux系统进行编译和运行

Is your feature request related to a problem? Please describe.
无法在arm架构下的linux系统进行编译

Describe the solution you'd like
Copying headers and resource to output directory.
/usr/bin/ld: ../../src/third_party/vnn/libs/linux/libvnn_core.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
make[2]: *** [src/CMakeFiles/gpupixel.dir/build.make:1832:../output/library/linux/libgpupixel.so] 错误 1
make[1]: *** [CMakeFiles/Makefile2:142:src/CMakeFiles/gpupixel.dir/all] 错误 2
make: *** [Makefile:84:all] 错误 2

Describe alternatives you've considered
在third_party/vnn/libs/linux的库是x86架构的,是否增加arm架构的,或者放一下源码路径,自己编译

Additional context
Add any other context or screenshots about the feature request here.

Clion Windows build segfall

building examples/desktop via CLion on Windows

I can successfully build application(after building the library)

When running with demo image everything is ok, but when I try to load another image instead of your demo.jpg file, I get Segfault at the line gpuSourceImage = SourceImage::create("other_img.jpg").

After some reseach it was found out that segfault occurs in the SourceImage::init function when trying to call glTexImage2D in line 60 when loading image into opengl.

CHECK_GL(glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, width, height, 0, GL_RGB,

can you tell me what to do to fix this error? not much experience with opengl

thanks

[BUG] App Compile on Windows

# Generate project
cmake -G "MinGW Makefiles" -B examples/build -S examples

# Should go to the examples folder instead of gpupixel
cd examples

# Build
cmake --build build -DCMAKE_BUILD_TYPE=Debug #or Release

VNN人脸关键点数据问题

人脸关键点检测使用VNN后,关键点的下标和 Face++是不一样的吧,我看最新代码使用的应该是 104 个关键点模式:
~~~
VNN_Set_Face_Attr(vnn_handle_, "_use_278pts", &use_278pts);
~~~
从 VNN 文档来看104 个关键点分布如下:
face_landmark_104pt
face++的 106 个关键点分布如下:
16ee5a14c5b24a47~tplv-t2oaga2asx-jj-mark_3024_0_0_0_q75
可以看到两种标注的下标是不一样的。

使用 VNN 后在美妆时getFaceIndexs、faceTextureCoordinates并没有同步修改为 VNN 标注的点,LipstickFilter的特效是如何生效的呢?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.