How does it work?

[rubyFeedback]

How does it work?

[woctezuma]

Just check the code. The gist of the algorithm is xuebinqin/U-2-Net.

backgroundremover/src/backgroundremover/bg.py

Lines 183 to 189 in 5cfa5bd

	img = Image.open(io.BytesIO(data)).convert("RGB")
	mask = detect.predict(model, np.array(img)).convert("L")
	if alpha_matting:
	cutout = alpha_matting_cutout(
	img,
	mask,

---

1. a segmentation mask is predicted with a neural network:
   • trained by Qin et al. "U2-Net: Going deeper with nested U-structure for salient object detection" Pattern Recognition, 2020,
   • available in xuebinqin/U-2-Net,

The first models (u2netp and u2net) are for salient object detection, and the other (u2net_human_seg) is for human segmentation.

backgroundremover/src/backgroundremover/u2net/detect.py

Lines 20 to 30 in 5cfa5bd

	'u2netp': (u2net.U2NETP,
	'e4f636406ca4e2af789941e7f139ee2e',
	'1rbSTGKAE-MTxBYHd-51l2hMOQPT_7EPy',
	'U2NET_PATH'),
	'u2net': (u2net.U2NET,
	'09fb4e49b7f785c9f855baf94916840a',
	'1ao1ovG1Qtx4b7EoskHXmi2E9rp5CHLcZ',
	'U2NET_PATH'),
	'u2net_human_seg': (u2net.U2NET,
	'347c3d51b01528e5c6c071e3cff1cb55',
	'1-Yg0cxgrNhHP-016FPdp902BR-kSsA4P',

---

2. then alpha-matting cut-out is performed using:
   • either a "naive" approach (just use the mask!),
   • or a higher-level one: morphological operations, followed by:
     • Germer et al. "Fast multi-level foreground estimation", 2020,
     • as implemented by the authors themselves in pymatting/pymatting as pymatting.foreground.

backgroundremover/src/backgroundremover/bg.py

Lines 158 to 161 in 5cfa5bd

	def naive_cutout(img, mask):
	empty = Image.new("RGBA", (img.size), 0)
	cutout = Image.composite(img, empty, mask.resize(img.size, Image.LANCZOS))
	return cutout

backgroundremover/src/backgroundremover/bg.py

Lines 132 to 133 in 5cfa5bd

	is_foreground = binary_erosion(is_foreground, structure=structure)
	is_background = binary_erosion(is_background, structure=structure, border_value=1)

backgroundremover/src/backgroundremover/bg.py

Line 7 in 5cfa5bd

from pymatting.foreground.estimate_foreground_ml import estimate_foreground_ml

---

On a side-note, the README was updated 3 days ago (5cfa5bd) with a link to a little blogpost mentioning U2-Net, which was posted two days before the OP asked the question. So the information was already available through this other mean but not as easily available as it is now.

---

Finally, I will mention a competitor to U2-Net, which is MODNet. I don't know which performs better.

• Ke et al. "Is a Green Screen Really Necessary for Real-Time Portrait Matting?", 2020
• ZHKKKe/MODNet
• the official online demo

[TheJackiMonster]

I assume it scans the image for a human, segments the image in parts which are connected to the potentially moving human in the foreground and other parts will be the background via exclusion. Otherwise it would probably not work for animated backgrounds.

Another approach would be using an additional image source of the background only to help separating between background and foreground via differences to the background source. But because there is no such additional source given as parameter in the examples, that doesn't work here.

The most consistent way would probably be an additional input from a depth sensor/camera. So there wouldn't be the requirement of a human or specific object to be recognized in the foreground. However then you had bigger hardware limitations for users of the software. ^^'

[nadermx]

Thank you @woctezuma I will also add, that the video part, has a whimsical with a video made by https://github.com/ecsplendid whom is the developer who's code I modified to get it working on this, which all came from this issue in the rembg library

[nadermx]

@woctezuma I'll also put on the todo to be able to import in different models.