Background Segmentation

DeepAR SDK uses deep neural networks to find and segment the background in the camera feed. The result of the Background segmentation is the background mask texture which can be used to change the background.

Background segmentation can be used concurrently with face tracking.

This feature enables the user to change the background of a photo or a video in any way they want.

Segmentation features
Background segmentation features are supported on iPhone7 and newer devices.

Example tutorial

In this example, we will demonstrate how to implement Background segmentation.

Background segmentation before and after

First what you need to do is create two new node quads to separate the foreground part from the background.

  • Open the DeepAR Studio

  • Now create a new effect via File → Create a new effect

  • Select the Root node, set the Position mode to "Position in camera space"

Choosing Position in camera space the object will NOT track the face, the object will have a fixed position, rotation and scale relative to the camera.

  • Next, select the Root node and add quad via NodeAdd Quad

  • Select new Quad, click NodeRename. Name the new quad "Background"

  • In Node properties set the layer of Background to Overlay

Next, do the same steps for another quad - Foreground Quad

  • Select the Root node and add quad via NodeAdd Quad

  • Select new Quad, click NodeRename. Name the new quad "Foreground"

  • In Node properties set the layer of Foreground to Overlay and Depth to 1 to ensure the Foreground is rendered above the Background

For the textures section, you will need to add two texture files. First is Foreground Mask, which is already added in DeepAR studio and the second is texture file which you need to add by yourself. Foreground Mask is there to help you cut the foreground part from the background. The texture file which you will add by yourself is the texture of what you want to be your background.


First, go to the Textures tab → Add → Foreground Mask

  • Next, add the new texture, in this case, we will add texture from the example folder (Background_segmentation_example/galaxy.jpg)

  • The texture tab should now look like this

Next, we are adding those textures to materials.

  • Select the Background quad

  • In the Materials properties disable the "Depth write" and "Depth test" options.

  • For Blend mode select Alpha

Disabling these two parts you will prevent the two quads from interfering with each other's rendering.

  • Under Select shader, drop-down lists choose "Unlit texture"

  • For Diffuse texture choose texture file, in this case, "galaxy.jpg"

The diffuse component of the material can be either set by the "Diffuse color" parameter in the basic "Standard shader" version, or by diffuse texture in the other two versions of the standard shader. This component defines the color of the material.

Next, we will set up materials for the foreground part

  • Select the Foreground quad

  • In the Materials properties disable the "Depth write" and "Depth test" options again

  • For Blend mode select Alpha

  • Under Select shader drop-down lists choose "Unlit texture Alpha"

  • For Diffuse texture choose "Camera texture"

  • For Alpha texture choose "Foreground mask"

To correctly scale the textures do the next things:

  • Select the Background quad

  • In the menu bar under Add Components select the Keep Texture Aspect Ratio component

Keep Texture Aspect Ratio - maintains the aspect ratio of textures applied to the node. Any scale values entered in the node transform will be multiplied with the calculated scale.

  • Select the Foreground quad

  • In the menu bar under Add Components select the Camera Quad component

Camera Quad - Scales the quad so the aspect ratio matches the camera feed.

And here is your final result

Download this example with texture assets

Did this answer your question?