UI explained

The DeepAR Studio is used to create effects usable from the DeepAR SDK and provides real-time feedback how the effect will look in the mobile app. Every parameter change will be reflected immediately in the live preview window.

DeepAR Studio UI with loaded effect

Besides the standard OSX options, the menu bar menu includes several other functionalities.

Preferences

Selecting DeepAR → Preferences will display the preferences popup.

The Import Settings tab allows the user to change the (BX animation sampling delta time. The default value is 40 ms, to reduce file size increase this value, and for very fast-moving detailed animations reduce this value. This will cause the newly imported animations to be sampled less frequently.

The View settings tab has the option to print out landmark values or stats on the screen for debugging purposes

DeepAR Studio preferences

Model loading

DeepAR Studio supports several ways of creating an effect:

DeepAR Studio menu bar

Selecting File → Create new effect DeepAR studio will create an empty effect with one root node in the scene

User can load previously exported effects with DeepAR studio by selecting File → Open effect. Files created with an old version of the Studio will be automatically converted.

Importing FBX models - From the menu bar select File → Import FBX and then choose a valid .fbx file. DeepAR Studio will try to import the FBX model and if successful the model will be displayed in the scene, otherwise an error message will be shown.

File → Import Legacy FBX scales the FBX files created for an old version of the Studio to work with the new face scale.

DeepAR Studio loads geometry data found in the .fbx file including all the meshes and vertices. Following parameters are imported for each vertex if they are available:

  • Position

  • Color

  • UV

  • Normal

  • Tangent

These parameters can be used in custom shader computations. DeepAR Studio loads FBX material data for each mesh/geometry. Material data can be additionally edited and refined in the DeepAR Studio, but it is important to add the material in the .fbx model itself. Currently, it is not possible to add additional materials with DeepAR Studio.

Once the user is done with model customization it must be exported in the file format that can be used by DeepAR SDK. To do so, from the menu bar choose File → Export model. In the next dialog choose a name and location to export the model. Additionally, DeepAR Studio can load models exported in this way and make further customizations. Newly created files will not work with Studio or SDK versions older than 2.0.

Adding models

Once a model is loaded from an exported file or created by any other method, another model can be added to the effect by selecting File → Add FBX or File → Import Legacy FBX for old models, to add a .fbx file or File → Add Studio File to add a file created in Studio. A new node will be added as a child of RootNode, this new node will be the original root node of the newly added file, will have the same name as the added file, and will also have it's own Animation Controller named the same as the node and the added file. Otherwise, the loading behaves the same as described in Model loading.

Material properties loading

Depending on the complexity of the model and the shaders used for the materials, it can be tedious to set all the parameters manually (as will be shown in subsequent sections). File → Import materials option allows users to import material property settings from one model to the current one in the scene. If two models share material with the same name then the properties will be overwritten from the one loaded with this option onto the currently loaded model. Material properties loading works only with files created with DeepAR Studio. Importing materials from old versions of the Studio might cause some settings to be imported incorrectly.

Assets menu

In the assets menu, the user has access to the custom shaders functionality and the animations functionality. Animations state editor can be used to control FBX model, image sequence, and sound animations.

Global lighting editor

When there are many materials defined with standard shader variants it can be tedious to set up all the lighting parameters for all the materials individually. The lighting editor panel allows setting lighting parameters globally. This means when we set up all the lighting parameters every material in the scene that has any variant of standard shader chosen will have lighting parameters set to the global ones in the Lighting editor panel.

DeepAR Studio global lighting editor

Node menu

The Node menu contains the options to edit the node hierarchy by adding, deleting, relocating, or renaming nodes in the node hierarchy.

DeepAR Studio node menu

Add Empty - creates an empty node. To add an empty node, select a node in the node hierarchy panel. The new node will be added as a child of the selected node.

Add Quad - creates a new quad. To add a quad, select a node in the node hierarchy panel. The new quad will be added as a child of the selected node.

Delete - removes the selected node and all its children from the hierarchy, deletes the underlying data as well as any materials that were only used by the deleted nodes

Move - opens a popup where the user can select a new parent for the currently selected node from a drop-down menu. The node's parent cannot be itself or one of its children. Only viable nodes will be listed. Nodes can also be moved by dragging and dropping in the Node hierarchy tree

Rename - opens a popup where the user can select a new name for the currently selected node. The RootNode cannot be renamed.

Add component menu

The Add component menu contains the components that can be added to the currently selected node. Components provide nodes with additional functionality or behavior.

DeepAR Studio Add Component menu

Face Position - Positions the node on the face

  • Ignore Translations - Ignore face translations.

  • Deprecated - cannot be used with individual ignoring individual axes or thresholds.

  • Ignore Rotations - Ignore face rotations.

  • Deprecated - cannot be used with individual ignoring individual axes or thresholds.

  • Ignore Translation X - Ignore face translation on the X-axis.

  • Ignore Translation Y - Ignore face translation on the Y-axis.

  • Ignore Translation Z - Ignore face translation on the Z-axis.

  • Ignore Rotations X - Ignore face rotation around the X-axis.

  • Ignore Rotations Y - Ignore face rotation around the Y-axis.

  • Ignore Rotations Z - Ignore face rotation around the Z-axis.

  • Threshold X - Value of the maximum absolute value allowed for X translation. Translations over the threshold are ignored. If the threshold is negative or zero, no threshold is applied.

  • Threshold Y - Value of the maximum absolute value allowed for Y translation. Translations over the threshold are ignored. If the threshold is negative or zero, no threshold is applied.

  • Threshold Z - Value of the maximum absolute value allowed for Z translation. Translations over the threshold are ignored. If the threshold is negative or zero, no threshold is applied.

  • Face ID - ID of the tracked face.

Keep Texture Aspect Ratio - maintains the aspect ratio of textures applied to the node, Any scale values entered in the node transform will be multiplied with the calculated scale.

  • Aspect Fit - Scales the node to fit the size of the view by maintaining the aspect ratio.

  • Aspect Fill - Scales the node to fill the size of the view by maintaining the aspect ratio - Default.

  • Aspect X - Scales the node to fit the width of the view by maintaining the aspect ratio.

  • Aspect Y - Scales the node to fit the height of the view by maintaining the aspect ratio.

  • Anchor Bottom - Changes the nodes anchor point to be at the bottom. The position can only be changed through a parent node. Can be active along with Anchor Left or Anchor Right

  • Anchor Top - Changes the nodes anchor point to be at the top. The position can only be changed through a parent node. Can be active along with Anchor Left or Anchor Right

  • Anchor Left - Changes the nodes anchor point to be on the left. The position can only be changed through a parent node. Can be active along with Anchor Bottom or Anchor Top

  • Anchor Right - Changes the nodes anchor point to be on the right. The position can only be changed through a parent node. Can be active along with Anchor Bottom or Anchor Top

Position At Landmark By Name - this component will have an effect if added on a node which is called jXX where XX is the number of the bone in the Reference model bones list. Node object will be driven by XX bone in the Reference model.

Play Animation on Mouth Open - Plays an animation on mouth open detection. The animation is stopped immediately as soon as the mouth close has been detected.

Play Uninterrupted Animation on Mouth Open - Plays an animation on mouth open detection. When the mouth is closed, the animation will not stop until the loop has finished.

Play Animation Continuous- plays an animation in an endless loop

Play Animation Once - plays an animation once

Camera Quad - Scales the quad so the aspect ratio matches the camera feed

Simple Pendulum Physics - Adds simple pendulum physics simulation to a node. For realistic results add this component to both ends of a pendulum.

Png Seq. with Animation - Ensures the png sequence plays at the desired frame rate

  • Frame Rate - controls the frame rate of the png sequence

Disable Child Nodes - Disables the children of the node it's attached to. Can be set to only disable children if a face is visible or not visible.

Emotions - Applies the detected emotion values to blend shapes or the corresponding name. The game object must have a mesh and a blend shape named: neutral, happiness, surprise, sadness, or anger.

Color Sampling - Allows the vertices to be colored with a sampled color. Requires a mesh that has the color sampling data encoded into its vertex colors. The source vertex should have the color R > 0, the target vertices should have R = 0 and their GB values should match. It also requires a custom shader that reads vertex colors.

Shader Time - Sets time uniform (u_currentTime) for shaders that use it

LeftEyePosition - Positions the object on the left pupil

Right Eye Position - Positions the object on the right pupil.

Toolbar

The Toolbar is displayed at the top of the DeepAR window and provides easy access to controls and information.

DeepAR Studio toolbar

The toolbar contains the following elements:

Play/Pause button

Resumes/Pauses camera feed. Can be used to freeze on a certain frame when using a webcam or to reduce processing costs.

Frame source

Allows the user to select the source for the tracking image. Available options are

  • Image (default) - uses default still image

  • Camera devices - as many as are available, uses the live feed from the selected camera

  • Image (from file) - browse to any still image.

Filename

Displays the name of the current effect.

Select aspect ratio

Resizes view to an aspect ratio from presets

  • Fit to window

  • iPhone portrait (9:16)

  • iPhone landscape (16:9)

  • iPhoneX portrait (9:19.5)

  • iPhoneX landscape (19.5:9)

Reload

Reloads current effect.


Node hierarchy tree

The node hierarchy panel provides a hierarchical view of model nodes. To edit certain properties of the node and meshes inside those nodes, the user needs to select the node in the hierarchy. A node doesn't necessarily have a visual representation in the scene, it can just be a container for other nodes and scene objects. Usually, the leaf node elements are objects like meshes that are visible in the scene or bone objects belonging to a skeleton. We usually set the transformation properties on parent nodes to position a group of objects in 3D space. That's why they are marked with a coordinate space icon. The leaf nodes, on the other hand, contain geometry and are marked with a 3D cube icon.

Nodes can be moved to a different parent by dragging and dropping.

DeepAR Studio node hierarchy

Node properties panel

When a node is selected, the right-hand view will display the widgets that pertain to it. All nodes have Transformations and Node Properties widgets.

DeepAR Studio node properties

Transformations

The Transformations widget allows a user to position the nodes in 3D space by editing position, rotation, and scale parameters in the x, y, and z-axis. Transform parameters can be set on parent nodes or on child nodes. Editing transformation parameters on a parent result in moving/rotating/scaling all of its children by the same amount.

DeepAR Studio transform toolbox

Node properties

In this widget, the user can edit certain properties that are important for nodes in the hierarchy. Those include:

  • Enabled

  • Enable animation

  • Position mode

  • Mesh driver

  • Dynamic UVs

  • Layer

  • Depth

DeepAR Studio node properties

Enabled

If the node is disabled it is removed from the rendering process along with all its children. The node is not removed from the hierarchy, but the engine skips the processing of the node if the value is set to false. The default value is true.

Enable animation

If animation is disabled on a node, transform and blend shape animations are not applied to this node. Default value is true.

Position mode

The position mode determines where the object will be placed. This property can only be set on the root node. There are the following options:

  • Position in face space: the object will track the face - it will match the translation and rotation of the face

  • Position in camera space: the object will NOT track the face - it will have a fixed position, rotation, and scale relative to the camera.

Mesh Driver

Mesh Driver is a property that is used to select a mechanism that drives the movement of the mesh. This property can be set for nodes that contain 3D mesh/geometry. Here are the possible options: Vertices Drivers can only be selected when using the corresponding Deformable meshes.

  • None - The mesh is static and is not driven at all

  • Bones - The mesh is driven by DeepAR face bones

  • Vertices 3D - Face - The mesh is driven by DeepAR face model

  • Vertices 2D - Lips - Tracks lips. This is used with the special mesh that tracks lips with high accuracy (typically used for the makeup)

  • Vertices 2D - Eyes - Tracks eyes and eyebrows. This is used with the special mesh that tracks eyes with high accuracy (typically used for the makeup)

  • Vertices 2D - Eyes v2 - Tracks only eyes. This is used with the special mesh that tracks eyes with high accuracy (typically used for the makeup)

Dynamic UVs

Texture (UV) coordinates are dynamically calculated and can be used to apply Camera texture or a user-created Render texture to a deformed Mesh. The possible modes are:

  • Disabled

  • Camera texture UV1 - Project the first texture coordinate set of the mesh to the Camera texture

  • Camera texture UV2 - Project the second texture coordinate set of the mesh to the Camera texture, typically used in conjunction with alpha and lighting masks.

  • Render texture UV1 - Project the first texture coordinate set of the mesh to a Render texture

  • Render texture UV2 - Project the second texture coordinate set of the mesh to the Render texture, typically used in conjunction with alpha and lighting masks.

Selected value can be applied to all of the node's children recursively by clicking on the icon right of the combo box.

Layer

Nodes in the virtual scene can be placed in layers that allow the user to adjust the rendering order for each node. The higher the layer value for the node is, the later the rendering of a given node will occur. This provides the user with the basic z-ordering functionality for node objects. This is comparable to the layer objects in Photoshop or Gimp. Some layers such as the Postprocessing layer have special behaviors and should only be used for their intended purpose. Layers 0 and Overlay are orthographic and will behave differently than perspective layers.

Selected value can be applied to all of the node's children recursively by clicking on the icon right of the combo box.

Depth

Allows the user to adjust the rendering order within a single layer. Nodes with greater depth value will be rendered later. Depth has no bearing on nodes in different layers.

Selected value can be applied to all of the node's children recursively by clicking on the icon right of the text box.

Blend Shapes

Mesh nodes can have a blend shape deformer attached to them. Blend shapes available for the selected mesh will be visible in the "Blend shapes" section of the Node settings. For each selected blend shape from the combo box control, the user can set its blend shape weight with the slider control or by entering the exact weight value into the text field. The current weight value will be saved in the effect file and will be used as such.

DeepAR Studio Blend shape settings

Materials properties

The Materials widget allows the user to edit properties for the material attached to the currently selected node.

DeepAR Studio Material properties

States

The materials states can control the rendering behavior of the material.

  • RGB write: renderer will write to RGB color buffer

  • Alpha write: renderer will write to Alpha buffer.

  • Depth write: renderer will write to Depth buffer.

  • Depth test: every fragment's output depth value will be tested against the Depth buffer.

Blend modes

DeepAR studio supports all the standard blending mode options which are set as material parameters:

  • Off - blending turned off

  • Add

  • Alpha

  • Darken

  • Lighten

  • Multiple

  • Normal

  • Screen

  • Linear burn

Culling modes

Geometry face culling options in the DeepAR Studio:

  • Off - no culling

  • Clockwise - default

  • Counterclockwise

Shader editor

Each material needs a shader assigned which defines how it will be rendered. By default, all materials use Standard shader. For more info on available shaders and how to write your own check the Shaders chapter.

Components

All components attached to the currently selected node will be added as widgets. Components can be removed by clicking the Remove Component button. Any properties the component has can be changed here.

DeepAR Studio Face Postition component

Textures editor

The texture list panel allows the user to load any textures that will be applied to the model. DeepAR Studio, by itself, doesn't load any textures that are found in .fbx (stored internally or externally). This means that every texture must be loaded in the DeepAR Studio by selecting appropriate action in Texture editor button. A loaded texture can be deleted by hitting the "Delete" button when the texture is selected in the list. The DeepAR Studio provides a default Camera texture and a Post-processing texture for each model. The Camera texture can be applied to models and the content of that texture corresponds to the current camera frame. The Post-processing texture should be applied to postprocessing effects.

Aside from textures, DeepAR supports frame animations. Frame animations are defined with a series of .png images that can be played in the effect. For animations to work in DeepAR, the png image sequence must be in the same folder and every image should follow the naming convention - "frame000000.png" for the first frame, "frame000001.png" for the second frame and so on. String "frame" is followed by 6 digits which increase for every frame, starting with 0. To load the animation user should click "Add → Animation" button and select the folder that contains the png frames that follow the previously described naming convention. Frame animations will run at 30 FPS by default. The FPS value can be changed by selecting the animation in the list and entering the desired value in the FPS field. For example, the lower framerate for the same animation means that the animation will be played at a slower speed. Animations can be triggered in different ways, read more about it in the components chapter.

Texture types

The texture can have the following types:

  • Texture (Regular): texture used for diffuse color

  • Animation: animated image loop sequence

  • Cubemap: a texture that can be used for realistic reflections

  • Render texture: special textures that can be rendered to - one or more layers can be rendered to a render texture instead of directly to screen

  • Hair mask: a special texture used for hair segmentation. The pixels that contain hair will have alpha = 100%. Cannot be used at the same time as a Foreground mask. At this time foreground segmentation doesn't work in the Studio. The placeholder hair mask will contain the top half of the screen. By default, this feature is disabled, if you need this feature to be enabled, please contact the support.

  • Foreground mask: a special texture used for foreground segmentation. The pixels that contain the foreground (i.e. the user) will have alpha = 100%. Cannot be used at the same time as a Hair mask. At this time foreground segmentation doesn't work in the Studio. The placeholder foreground mask will contain the top half of the screen. By default, this feature is disabled, if you need this feature to be enabled, please contact the support.

  • Skin texture

DeepAR Studio texture list

Texture quality

When adding a new texture, cubemap or animation the user will be prompted to select the desired texture quality. The texture will then be encoded and stored with the selected settings. Texture quality cannot be changed once the texture is loaded. Available quality presets are:

  • PNG 100%

  • JPEG 100%

  • JPEG 80%

  • JPEG 60%

  • JPEG 40%

Only the PNG preset retains the alpha channel, whereas the JPEG presets set the texture alpha to fully opaque. JPEG presets produce textures that require significantly less memory to store and are faster to encode and decode. Selecting a lower percentage preset will reduce the image quality and further reduce the memory required to store the texture. Loss of image quality will be most apparent when working with stylized textures that have large areas covered in one color or gradients.

Render texture setup

Render textures are a special type of textures that are updated in the runtime. When creating a new render texture the user will be prompted to name the new render texture and select the layers that will be rendered to this texture. One or more layers can be rendered to a single render texture. A layer can only belong to one render texture. Layers that are rendered to render texture will not be visible on the screen.

DeepAR Studio render texture setup

Materials list

Materials list panel shows all the materials in a loaded effect.

DeepAR Studio materials list

Sounds editor

Sounds list panel allows the user to load any sounds that will be used by the effect, and remove already loaded sounds. The supported audio formats are .mp3 and .wav. Sounds are assigned to states in the Animation state editor.

DeepAR Studio sounds editor

Did this answer your question?