Go to DeepAR
All Collections
DeepAR Physics & Scripting
Face painting effect tutorial

Face painting effect tutorial

Face painting effect achieved using touch detection and scripting
Matej Trbara avatar
Written by Matej Trbara
Updated over a week ago

In this example, the usage of touch detection and touch-specific scripting features will be showcased by creating a real-time face painting effect. For further information on how to forward touch occurrences from the platform used to the DeepAR engine appropriately, please refer to the example applications linked at the end of the article.

Step 0

The given ZIP contains the unlit_texture_alpha_cutoff folder with a custom Unlit Texture Alpha Cutoff shader, written to support the eraser feature within the effect. The path to the extracted folder should be explicitly added within the Studio through Assets → Set shader paths.

The shader within the added folder can be compiled and added to the Studio by selecting Assets → Compile shaders → Compile shaders. For further shader assistance and questions, refer to this and this article.

Step 1 - face mesh import

Import the face mesh original_face.fbx file using File → Import FBX. The loaded scene contains a node named Mesh, which can optionally be renamed to FaceMesh (and it will be referred to as such in the rest of the article) by choosing the node and selecting Node → Rename.

Step 2 - paint brush node

Add a new quad node as the child node of the RootNode and name it PaintBrush. Any other name can be given, but it is important to be consistent with the name usage of this specific node, since the node will be accessed within scripting using its name.

At this point, the scene hierarchy should look like this:

Step 3 - paint brush texture

Instead of the default quad, a custom paint brush texture can be imported. Within Studio, choose the Texture tab, click Add → Texture and choose the supplied whiteCircle.png file (with PNG 100% quality and without creating mipmaps). The chosen color of the texture was white so that any color chosen as the material color in the following steps won't mesh with the initial color of the texture.

Step 4 - render texture

In order to display the painting done in real time onto the face mesh, a separate render texture should be created, in a similar way as in the previous step, but by selecting Add → RenderTexture. If not familiar with render textures, check out this and this article.

The render texture will be attached to the face mesh and will instantaneously display what is being painted by the user. The layer that is to be rendered to the render texture is the orthographic layer Overlay (with the layer index 11), and everything that should be rendered onto the face mesh should be placed in that layer.

The Textures tab should now contain two new textures:

Step 5 - RootNode modification

  • change Position mode within the Node properties component to Position in camera space

  • add the scripting component through Add Component → Script, which will later have the dedicated face painting script attached to it

Step 6 - PaintBrush modification

  • set the scale property to 0.03 for all three axes - initial size of the paint brush

  • change the layer value to the Overlay (11) layer

  • in the Material component of the node:

    • uncheck the Depth test property

    • set the Blend Mode property to Off

    • set the Culling Mode property to Off

    • select the Unlit Texture Alpha Cutoff shader

      • choose the previously added whiteCircle.png as the Diffuse and Alpha texture

      • set the Color to the desired paint brush color, in this example black

Step 7 - FaceMesh modification

  • add the Face Position component to the node by clicking Add Component → Script, since the root node is set to be positioned in camera space

  • in the Material component of the node:

    • set the Blend Mode property to Alpha

    • set the Culling Mode property to Clockwise

    • select the Unlit Texture & Color shader

      • choose the previously created RenderTexture (11) as the Diffuse map value

      • optionally, change the opacity of the entire face mesh through the Color property, in this example 75% opacity

Step 8 - scripting

For explanations of the used touch-specific and other scripting features within this script, please refer to the DeepAR Scripting API Documentation.

Create a new JavaScript file, here called facePaintingScript.js, and write a script used to draw the paint brush texture onto the intersected area of the screen. The script used for this example contains:

  • global variables

    • var deltaDefault = 0.5, delta1 = 0.25, delta2 = 0.01, threshold1 = 0.001 * 0.001, threshold2 = 0.05 * 0.05

      • delta values - the selected delta will be the value by which the interpolation parameter will be increased, therefore determining the amount of points drawn between two given intersections

      • threshold values - square values of distance thresholds required for specific delta assignment 

    • var rootNode

      • variable storing the RootNode scripting Node representation

    • var paintBrush, paintBrushNodeName = 'PaintBrush'

      • paintBrush - variable storing the PaintBrush scripting Node representation

    • var faceMesh, faceMeshNodeName = 'FaceMesh'

      • faceMesh - variable storing the FaceMesh scripting Node representation

    • var lastUV, lastTouchType, lastTouchIntersected

      • lastUV - Vector2 value containing information about the transformed UV values of the previous intersection

      • lastTouchType - Touch.TouchType value containing information about what the touch type of the previous intersection was

      • lastTouchIntersected - boolean value containing information about whether the previous touch that occurred was an intersection

    • var lastRenderResolution = new Vec2()

      • lastRenderResolution - Vector2 value containing the last known render resolution

    • var viewId = 11

      • index of the layer/view that is to be manipulated with, in this case the orthographic layer Overlay with the index 11

  • onStart() callback function

    • Utility.setViewCleared(viewId, false, false)

      • setting the flags for color and depth buffer clearing of the selected layer 

    • rootNode = Node.root
      faceMesh = rootNode.getChild(faceMeshNodeName)
      paintBrush = rootNode.getChild(paintBrushNodeName)

      • fetching the scripting Node representations of the nodes within the scene

    • paintBrush.enabled = false

      • initially disabling the paint brush node so that it does not show up on the screen before any occurred touches

    • lastTouchIntersected = false
      lastRenderResolution = Context.renderResolution

      • initialising the values required for further touch processing

  • onPreUpdate() callback function

    • var renderResolution = Context.renderResolution

      • fetching the current render resolution

    • if (!renderResolution.equals(lastRenderResolution)) {
          Utility.setViewCleared(viewId, true, true)
          lastRenderResolution = renderResolution
      }

      • checking whether the render resolution changed; if so, the layer containing the painting is cleared and the last know render resolution is updated

  • onTouchOccurred(touch, type) callback function

    • var intersection = faceMesh.performRayCasting(touch.screenPosition, false)

      • retrieving intersection information on a specific node

        • information forwarded:

          • screen position of the occurred touch, given as 2D screen coordinates

          • boolean value informing whether recursive ray casting should occur (taking the node's descendants into account)

    • if (intersection === null) 
          touchOutsideMeshOccurred()
      else
          intersectionOccurred(intersection, type)

      • if intersection is null, then no intersection occurred on the given node

      • if intersection is not null, then the intersection information, as well

  • touchOutsideMeshOccurred() function

    • if (paintBrush.enabled)
         paintBrush.enabled = false

      • disabling the paint brush node in order to signal touch occurring outside of the mesh

    • lastTouchIntersected = false

      • updating information about the previous touch intersection status

  • intersectionOccurred(intersection, type) function

    • Utility.setViewCleared(viewId, false, false)

      • setting the flags for color and depth buffer clearing of the selected layer

    • var uv = intersection.UVCoordinates.mul(2).sub(1).mul(new Vec2(1, -1))
      var zValue = paintBrush.localPosition.z

      • uv - respective UV coordinates of the 3D intersection on the mesh, transformed from range [0, 1] to [-1, 1] and mirrored along the y-axis so that the painting done by the user renders correctly

      • zValue - z-axis value of the current intersection, later necessary for proper 3D local positioning of the paint brush node, with already available 2D x and y values from the transformed UV coordinates

    • switch(type) {
          case Touch.TouchType.START:
              intersectionStart(uv, zValue)
              break
          case Touch.TouchType.MOVE:
              intersectionMove(uv, zValue)
              break
          case Touch.TouchType.END:
              intersectionEnd(uv, zValue)
              break
          default:
              Debug.log("unsupported touch type")
      }

      • determining which touch type occurred and performing appropriate actions accordingly

    • lastUV = uv
      lastTouchType = type
      lastTouchIntersected = true

      • updating information about the previous touch/intersection

  • intersectionStart(uv, zValue) function

    • Utility.setViewCleared(viewId, false, false)
      paintBrush.enabled = true

      • setting the flags for color and depth buffer clearing of the selected layer and enabling the paint brush node so that it is visible and "paintable" onto the layer

    • paintBrush.localPosition = new Vec3(uv, zValue)
      paintBrush.drawExplicitly()

      • updating the local position of the brush to the previously defined UV coordinates, while retaining the z value of the intersection, and initiating a draw call for the change that was made

  • intersectionMove(uv, zValue) function

    • Utility.setViewCleared(viewId, false, false)

      • setting the flags for color and depth buffer clearing of the selected layer (the paint brush node should already be enabled, since it is impossible to get to the TouchType.MOVE state without being in the TouchType.START state previously, when the paint brush node is enabled)

    • if (typeof lastUV === 'undefined') {
          paintBrush.enabled = true
              
          paintBrush.localPosition = new Vec3(uv, zValue)
          paintBrush.drawExplicitly()
      } else
          drawWithPossibleInterpolation(uv, zValue)

      • if lastUV is undefined, that means that the initial touch within the effect occurred outside of the face mesh and the user went over the mesh during the same dragging event, resulting in the first face mesh intersection to be of the TouchType.MOVE type and the lastUV not yet being set (making it not possible to interpolate between the last two UV values - the point is simply drawn onto the screen)

      • if lastUV is not undefined, then a previous face mesh intersection exists and it is possible to interpolate between the appropriate two UV values for the previous and current intersection

  • intersectionEnd(uv, zValue) function

    • if (!paintBrush.enabled)
          return

      • cancelling drawing the touch if the paint brush node had not yet been enabled, i.e, coming from the TouchType.START state entered when touching outside the face mesh to the TouchType.END state by intersecting the face mesh

    • drawWithPossibleInterpolation(uv, zValue)

      • forwarding information when the previous face mesh intersection exists and it is possible to interpolate between the appropriate two UV values for the previous and current intersection 

    • paintBrush.enabled = false

      • disabling the paint brush node (changing its visibility) after finishing actions related to the TouchType.END state

  • drawWithPossibleInterpolation(uv, zValue) function

    • if (lastTouchIntersected) {
          var distance = Math.sqrt(Math.pow(lastUV.x - uv.x, 2) + Math.pow(lastUV.y - uv.y, 2), 2)
                  
          delta = deltaDefault
          if (distance > threshold1)
              delta = (distance > threshold2) ? delta2 : delta1
                    
          for (var t = delta; t <= 1; t += delta) {
              paintBrush.localPosition = new Vec3(lerp(lastUV.x, uv.x, t), lerp(lastUV.y, uv.y, t), zValue)
              paintBrush.drawExplicitly()
          }
      } else {
          paintBrush.localPosition = new Vec3(uv, zValue)
          paintBrush.drawExplicitly()
      }

      • if statement - checking whether interpolation should occur, depending on whether the last touch was also an intersection - if so, the UV values of both the previous and current intersection interpolate with respect to a specific delta value, issuing more draw calls in between the two intersections, therefore creating a more consistent line

      • var distance = Math.sqrt(Math.pow(lastUV.x - uv.x, 2) + Math.pow(lastUV.y - uv.y, 2), 2)

        • computing the distance between two UV values

      • delta = deltaDefault

        if (distance > threshold1)

        delta = (distance > threshold2) ? delta2 : delta1

        • determining the delta value used for increasing the interpolation parameter based on the comparison between the calculated distance and predefined thresholds with accompanying delta values

          • if the distance between the UV values of the intersections are close enough, sparse interpolation will occur; otherwise, one of the smaller delta values will be used in order to provide more points that will be the result of the interpolation between the two intersections

      • for (var t = delta; t <= 1; t += delta) {

        paintBrush.localPosition = new Vec3(lerp(lastUV.x, uv.x, t), lerp(lastUV.y, uv.y, t), zValue)

        paintBrush.drawExplicitly()

        }

        • perform interpolation for every given parameter t ∈ [0, 1], t increasing every iteration for the given delta

  • lerp(x0, x1, t) function

    • return (1 - t) * x0 + t * x1

      • linear interpolation between values x0 and x1, given parameter t

The finished script (also available as a part of the downloadable ZIP) can be attached to the Script component of the RootNode.

Step 9 - test the effect

The effect should now be finished and ready to have fun with!

The extended usage of the painting effect, with features such as erasing or changing the color and size of the paint brush, can be tested by using the Face Painting example applications for iOS (Objective-C) and Android (Java).

Download the ZIP with all the necessary assets and the final effect

Download
Did this answer your question?