xiongzhu666/sky-segmentation-and-post-processing

Sky-Segmentation-and-Post-processing

This is a C++ implementation from this paper https://arxiv.org/abs/2006.10172 that published on 2020, the repo is for sky mask post-processing. but I didn't implemente the "Density Estimation" mentioned in the paper.

About Sky segmentation, I trained the sky-segmentation model by U-2-Net, the result looks good. please refer to https://github.com/xuebinqin/U-2-Net about training detail

Dependency：OpenCV, ncnn

seg_demo.cpp is for sky-seg and input is image

mask_refine.cpp is for mask post-process to refine the mask. inputs are image and the mask inferenced by model.

The Sky-mask Post-Processing show a good performence in the scene of tree as below. it retain much more details.In addition, the post-process is only for sky-mask.perhaps it won't get the same good performance when you apply it on other class segmentation.

2023/11/22 Update: EGE-unet and u2netp speed test

2021/12/29 Update: upload code interenced by onnxruntime, you need to install the package by pip install onnxruntime

onnx model(167M) baiduyun：https://pan.baidu.com/s/1bE38w422STSwuJwjPpRIMw code：4tmm

2021/10/13 Update

Upload a small sky-seg model of 2Mb（traind by u2netp） for demo（We couldn't public the high-precision model because it used in our product）

Upload a sky-seg demo cpp inferenced by ncnn

but it also has some defect：in the scene of building, some detail of building will be considered as sky by mistake

For some special textured clouds, The algorithm has some flaws as below

Next TODO: the U-2-Net couldn't run in real-time in mobile device(about 300ms in Snapdragon 888). even though u2netp size is much smaller than u2net, but the interence speed doesn't improve obviously. I plan to train a real-time model by normanl unet so that it could run in real-time in mobile device.

sky-segmentation-and-post-processing's Issues

Sky Segmentation Pretrain

Hi @xiongzhu666,
This was great work. Is available for Sky-Seg pretrained?

Quality difference between EGE-unet and u2netp

it seems that there is a significant quality difference in the output of EGE-unet and u2netp.,even in your test example. If it is in dark night sky, EGE-unet is completely unusable. If that is what expected, I think user may want to be noted in the README. Thanks.

u2netp:

EGE-unet:

How to use the non-binary refined mask and how do you upsample the mask

Could you tell us what paper or algorithm your mask_refine.ccp is based on

Could you tell us what the paper mentioned in your mask_refine.ccp and what algorithms it implements?

Thank you!

Issue downloading .ONNX

Hi,

Thanks for your work!
It seems like the link for the onnx file is broken.
Is there alternative?

Thanks

model在IOS上加载出错提示“parse magic failed”

Hi，

The code is

int SkySegmentation::load(const char* paramString, const char* binString, unsigned int _num_threads, int _input_width, int _input_height)
{
net.clear();
net.opt.use_vulkan_compute = false;
net.load_param(paramString);
net.load_model(binString);

input_width  = _input_width;
input_height = _input_height;
num_threads  = _num_threads;

return 0;

}

void SkySegmentation::transform(const cv::Mat &mat_rs, ncnn::Mat &in)
{
// BGR NHWC -> RGB NCHW
in = ncnn::Mat::from_pixels(mat_rs.data, ncnn::Mat::PIXEL_BGR2RGB, input_width, input_height);
in.substract_mean_normalize(mean_vals, norm_vals);
}

void SkySegmentation::detect(cv::Mat &rgb, cv::Mat &mask,)
{
if (rgb.empty()) return;
int img_height = static_cast(rgb.rows);
int img_width = static_cast(rgb.cols);

ncnn::Mat input;
this->transform(rgb, input);
ncnn::Extractor ex = net.create_extractor();
ex.set_light_mode(true);
ex.set_num_threads(4);
ex.input("input.1", input);

ncnn::Mat out;
ex.extract("1959", out);
//cv::Mat opencv_mask(out.h, out.w, CV_32FC1);
//memcpy((uchar*)opencv_mask.data, out.data, out.w * out.h * sizeof(float));
//cv::resize(opencv_mask,opencv_mask,cv::Size(w,h),cv::INTER_LINEAR);

float *pha_data = (float*)out.data;
cv::Mat cv_mask = cv::Mat::zeros(out.h, out.w, CV_8UC1);
cv::Mat cv_pha = cv::Mat(out.h, out.w, CV_32FC1, pha_data);

cv_pha.convertTo(cv_mask, CV_8UC1, 255.0, 0);
cv_mask.copyTo(mask);

}

当我运行上面的代码时提示 parse magic failed
加载model代码
NSString *parmaPath = [[NSBundle mainBundle] pathForResource:@"skysegsmall_sim-opt-fp16" ofType:@"param"];
NSString *binPath = [[NSBundle mainBundle] pathForResource:@"skysegsmall_sim-opt-fp16" ofType:@"bin"];
skySegmentation = new SkySegmentation;
skySegmentation->load([parmaPath UTF8String], [binPath UTF8String]);
我检查了代码发现在 net.load_param(binString); 时有问题。是不是.bin文件有问题？

我测试其他的model在demo上加载运行正常。

CUDA Out of Memory with while running Sky segmentation TRT model Inference on GPU A2000

Hello,

Thanks for sharing your work on Sky Segmentation.
I converted the ONNX model to TensorRT and I'm able to run inference on a Video but only for about 90 seconds.
Then I see CUDA Runtime Error 2 (Out of Memory)

Environment:

• TensorRT Version: 8.6.1.6
• NVIDIA GPU: A2000
• NVIDIA Driver Version: 510.39.01
• CUDA Version: 11.6
• CUDNN Version: 8.9.4
• Operating System: Ubuntu 20.04
• Python Version : 3.8.10

Steps To Reproduce:
Step1: Download the ONNX file from Sky-Segmentation and Post processing Git Repo or the link here

Step2: Convert ONNX to TRT using the repo TensorRT-Alpha

Step3: Run inference on a video using the script U2-Net TensorRT

Error Encountered after approx. 90 seconds:

Any idea what could have caused this?

Image resolution

Hi Xiong,

Recently I am reading that google paper, and I have a issue want to confirm with you.

As I see from the pipeline, in order to get coarse matte, we need to first down size the input raw image, And I think this down-sized image should be the input of matte inference network( for your case is U2net). and the following upsampled weighted guided filter is used to upsample the low resolution output from u2net to a high resolution same as input.

Am I correct?
With warm regards
Kangning Li

Python version guided filter\mask refine comparison

There is a python implementation which generates noticably different results(noticeable on the border, python version preserves more, while this repo version "damages" the original correct border)
https://github.com/google/sky-optimization/pull/14/files
Are there any major differences between your implementation and the python version and if so how could they be changed?

May I ask how can I convert this ncnn model to pytorch model