Deformable masks

Deformable masks are driven by facial expressions. They can use full 3D vertices, a limited but sometimes more precise set of 2D vertices, or bones. For information on how to use deformable masks see the Mesh Driver section in UI explained.

3D vertices

Models driven by 3D vertices should be placed in face space. Vertex colors or the number of vertices should not be changed.

Reference mask model

The reference mask model is the underlying model that is driven by DeepAR face-tracking technology. Users of the SDK can use it as a starting point in creating their own effects in 3D modeling tools (like Blender or Maya). Below is the image of the reference model imported in Blender.

Reference model imported in Blender

Reference model vertex color values

The reference model has a geometry that consists of 730 vertices, all of which are driven by tracked 3D face vertices.

Each vertex in a mesh can store an RGB color value. The engine uses this feature to differentiate every vertex by its unique color property and match it to the tracked feature point on the face.

The vertex colors have already been set in the provided reference model and shouldn't be changed or omitted when creating a new model - otherwise, the engine will not be able to track a given face feature point and the mesh will not display correctly when imported.

To check whether the vertex colors have been preserved, import the newly created .fbx model in Blender, select the face mesh, and toggle Vertex Paint. It should look the same as the image above.

2D vertices

Models driven by 2D vertices should be placed in camera space. Vertex colors, or the number of vertices should not be changed.

Eyes

There are two eye models provided. The first model consists of 304 vertices and tracks the vertices surrounding the eyes and eyebrows. The second model consists of 192 vertices and does not track eyebrows.

These models can be used for a more precise eyelid occlusion than the 3D vertices provide. The vertex colors aver been set in the eyes models and matched to the tracked feature points on the face.

Eyes and eyebrows model vertex color values

Lips

The lips model consists of 232 vertices driven by tracked 2D face vertices.

This model can be used for more precise lips tracking than the 3D vertices provide. The vertex colors aver been set in the lips model and matched to the tracked feature points on the face.

Lips model vertex color values


Bones

Other than vertices, user-created masks can be driven by bones. Bones are used when the model isn't created from a reference model. In that case, it is easier to map significant points of interest in reference to the resulting model since the number of bones is smaller than the number of vertices. The user doesn't have to define every bone in the resulting model. Bones are defined by their names which is in the format of "jxx" where the "xx" part is the number of the bone in the reference model. Below is the map of bones available in reference model:

Reference model bones


Here you can download the fbx examples:

  • The original face mesh as an example of a model driven by 3D vertices

  • The eyes and lips meshes as an example of models driven by 2D vertices

  • The default bone rig as an example of a bone rig model with bones

NOTE: Make sure to use Import Legacy Fbx option when loading the provided .fbx files for the original face and dense mesh.

Did this answer your question?