In this example, the usage of touch detection and touch-specific scripting features will be showcased by creating a real-time face painting effect. For further information on how to forward touch occurrences from the platform used to the DeepAR engine appropriately, please refer to the example applications linked at the end of the article.
Step 0
The given ZIP contains the unlit_texture_alpha_cutoff folder with a custom Unlit Texture Alpha Cutoff shader, written to support the eraser feature within the effect. The path to the extracted folder should be explicitly added within the Studio through Assets → Set shader paths.
The shader within the added folder can be compiled and added to the Studio by selecting Assets → Compile shaders → Compile shaders. For further shader assistance and questions, refer to this and this article.
Step 1 - face mesh import
Import the face mesh original_face.fbx file using File → Import FBX. The loaded scene contains a node named Mesh, which can optionally be renamed to FaceMesh (and it will be referred to as such in the rest of the article) by choosing the node and selecting Node → Rename.
Step 2 - paint brush node
Add a new quad node as the child node of the RootNode and name it PaintBrush. Any other name can be given, but it is important to be consistent with the name usage of this specific node, since the node will be accessed within scripting using its name.
At this point, the scene hierarchy should look like this:
Step 3 - paint brush texture
Instead of the default quad, a custom paint brush texture can be imported. Within Studio, choose the Texture tab, click Add → Texture and choose the supplied whiteCircle.png file (with PNG 100% quality and without creating mipmaps). The chosen color of the texture was white so that any color chosen as the material color in the following steps won't mesh with the initial color of the texture.
Step 4 - render texture
In order to display the painting done in real time onto the face mesh, a separate render texture should be created, in a similar way as in the previous step, but by selecting Add → RenderTexture. If not familiar with render textures, check out this and this article.
The render texture will be attached to the face mesh and will instantaneously display what is being painted by the user. The layer that is to be rendered to the render texture is the orthographic layer Overlay (with the layer index 11), and everything that should be rendered onto the face mesh should be placed in that layer.
The Textures tab should now contain two new textures:
Step 5 - RootNode modification
change Position mode within the Node properties component to Position in camera space
add the scripting component through Add Component → Script, which will later have the dedicated face painting script attached to it
Step 6 - PaintBrush modification
set the scale property to 0.03 for all three axes - initial size of the paint brush
change the layer value to the Overlay (11) layer
in the Material component of the node:
uncheck the Depth test property
set the Blend Mode property to Off
set the Culling Mode property to Off
select the Unlit Texture Alpha Cutoff shader
choose the previously added whiteCircle.png as the Diffuse and Alpha texture
set the Color to the desired paint brush color, in this example black
Step 7 - FaceMesh modification
add the Face Position component to the node by clicking Add Component → Script, since the root node is set to be positioned in camera space
in the Material component of the node:
set the Blend Mode property to Alpha
set the Culling Mode property to Clockwise
select the Unlit Texture & Color shader
choose the previously created RenderTexture (11) as the Diffuse map value
optionally, change the opacity of the entire face mesh through the Color property, in this example 75% opacity
Step 8 - scripting
For explanations of the used touch-specific and other scripting features within this script, please refer to the DeepAR Scripting API Documentation.
Create a new JavaScript file, here called facePaintingScript.js, and write a script used to draw the paint brush texture onto the intersected area of the screen. The script used for this example contains:
global variables
var deltaDefault = 0.5, delta1 = 0.25, delta2 = 0.01, threshold1 = 0.001 * 0.001, threshold2 = 0.05 * 0.05delta values - the selected delta will be the value by which the interpolation parameter will be increased, therefore determining the amount of points drawn between two given intersections
threshold values - square values of distance thresholds required for specific delta assignment
var rootNodevariable storing the RootNode scripting Node representation
var paintBrush, paintBrushNodeName = 'PaintBrush'paintBrush - variable storing the PaintBrush scripting Node representation
var faceMesh, faceMeshNodeName = 'FaceMesh'faceMesh - variable storing the FaceMesh scripting Node representation
var lastUV, lastTouchType, lastTouchIntersectedlastUV - Vector2 value containing information about the transformed UV values of the previous intersection
lastTouchType - Touch.TouchType value containing information about what the touch type of the previous intersection was
lastTouchIntersected - boolean value containing information about whether the previous touch that occurred was an intersection
var lastRenderResolution = new Vec2()lastRenderResolution - Vector2 value containing the last known render resolution
var viewId = 11index of the layer/view that is to be manipulated with, in this case the orthographic layer Overlay with the index 11
onStart() callback function
Utility.setViewCleared(viewId, false, false)setting the flags for color and depth buffer clearing of the selected layer
rootNode = Node.root
faceMesh = rootNode.getChild(faceMeshNodeName)
paintBrush = rootNode.getChild(paintBrushNodeName)fetching the scripting Node representations of the nodes within the scene
paintBrush.enabled = falseinitially disabling the paint brush node so that it does not show up on the screen before any occurred touches
lastTouchIntersected = false
lastRenderResolution = Context.renderResolutioninitialising the values required for further touch processing
onPreUpdate() callback function
var renderResolution = Context.renderResolutionfetching the current render resolution
if (!renderResolution.equals(lastRenderResolution)) {
Utility.setViewCleared(viewId, true, true)
lastRenderResolution = renderResolution
}checking whether the render resolution changed; if so, the layer containing the painting is cleared and the last know render resolution is updated
onTouchOccurred(touch, type) callback function
var intersection = faceMesh.performRayCasting(touch.screenPosition, false)retrieving intersection information on a specific node
information forwarded:
screen position of the occurred touch, given as 2D screen coordinates
boolean value informing whether recursive ray casting should occur (taking the node's descendants into account)
if (intersection === null)
touchOutsideMeshOccurred()
else
intersectionOccurred(intersection, type)if intersection is null, then no intersection occurred on the given node
if intersection is not null, then the intersection information, as well
touchOutsideMeshOccurred() function
if (paintBrush.enabled)
paintBrush.enabled = falsedisabling the paint brush node in order to signal touch occurring outside of the mesh
lastTouchIntersected = falseupdating information about the previous touch intersection status
intersectionOccurred(intersection, type) function
Utility.setViewCleared(viewId, false, false)setting the flags for color and depth buffer clearing of the selected layer
var uv = intersection.UVCoordinates.mul(2).sub(1).mul(new Vec2(1, -1))
var zValue = paintBrush.localPosition.zuv - respective UV coordinates of the 3D intersection on the mesh, transformed from range [0, 1] to [-1, 1] and mirrored along the y-axis so that the painting done by the user renders correctly
zValue - z-axis value of the current intersection, later necessary for proper 3D local positioning of the paint brush node, with already available 2D x and y values from the transformed UV coordinates
switch(type) {
case Touch.TouchType.START:
intersectionStart(uv, zValue)
break
case Touch.TouchType.MOVE:
intersectionMove(uv, zValue)
break
case Touch.TouchType.END:
intersectionEnd(uv, zValue)
break
default:
Debug.log("unsupported touch type")
}determining which touch type occurred and performing appropriate actions accordingly
lastUV = uv
lastTouchType = type
lastTouchIntersected = trueupdating information about the previous touch/intersection
intersectionStart(uv, zValue) function
Utility.setViewCleared(viewId, false, false)
paintBrush.enabled = truesetting the flags for color and depth buffer clearing of the selected layer and enabling the paint brush node so that it is visible and "paintable" onto the layer
paintBrush.localPosition = new Vec3(uv, zValue)
paintBrush.drawExplicitly()updating the local position of the brush to the previously defined UV coordinates, while retaining the z value of the intersection, and initiating a draw call for the change that was made
intersectionMove(uv, zValue) function
Utility.setViewCleared(viewId, false, false)setting the flags for color and depth buffer clearing of the selected layer (the paint brush node should already be enabled, since it is impossible to get to the TouchType.MOVE state without being in the TouchType.START state previously, when the paint brush node is enabled)
if (typeof lastUV === 'undefined') {
paintBrush.enabled = true
paintBrush.localPosition = new Vec3(uv, zValue)
paintBrush.drawExplicitly()
} else
drawWithPossibleInterpolation(uv, zValue)if lastUV is undefined, that means that the initial touch within the effect occurred outside of the face mesh and the user went over the mesh during the same dragging event, resulting in the first face mesh intersection to be of the TouchType.MOVE type and the lastUV not yet being set (making it not possible to interpolate between the last two UV values - the point is simply drawn onto the screen)
if lastUV is not undefined, then a previous face mesh intersection exists and it is possible to interpolate between the appropriate two UV values for the previous and current intersection
intersectionEnd(uv, zValue) function
if (!paintBrush.enabled)
returncancelling drawing the touch if the paint brush node had not yet been enabled, i.e, coming from the TouchType.START state entered when touching outside the face mesh to the TouchType.END state by intersecting the face mesh
drawWithPossibleInterpolation(uv, zValue)forwarding information when the previous face mesh intersection exists and it is possible to interpolate between the appropriate two UV values for the previous and current intersection
paintBrush.enabled = falsedisabling the paint brush node (changing its visibility) after finishing actions related to the TouchType.END state
drawWithPossibleInterpolation(uv, zValue) function
if (lastTouchIntersected) {
var distance = Math.sqrt(Math.pow(lastUV.x - uv.x, 2) + Math.pow(lastUV.y - uv.y, 2), 2)
delta = deltaDefault
if (distance > threshold1)
delta = (distance > threshold2) ? delta2 : delta1
for (var t = delta; t <= 1; t += delta) {
paintBrush.localPosition = new Vec3(lerp(lastUV.x, uv.x, t), lerp(lastUV.y, uv.y, t), zValue)
paintBrush.drawExplicitly()
}
} else {
paintBrush.localPosition = new Vec3(uv, zValue)
paintBrush.drawExplicitly()
}if statement - checking whether interpolation should occur, depending on whether the last touch was also an intersection - if so, the UV values of both the previous and current intersection interpolate with respect to a specific delta value, issuing more draw calls in between the two intersections, therefore creating a more consistent line
var distance = Math.sqrt(Math.pow(lastUV.x - uv.x, 2) + Math.pow(lastUV.y - uv.y, 2), 2)computing the distance between two UV values
delta = deltaDefaultif (distance > threshold1)delta = (distance > threshold2) ? delta2 : delta1determining the delta value used for increasing the interpolation parameter based on the comparison between the calculated distance and predefined thresholds with accompanying delta values
if the distance between the UV values of the intersections are close enough, sparse interpolation will occur; otherwise, one of the smaller delta values will be used in order to provide more points that will be the result of the interpolation between the two intersections
for (var t = delta; t <= 1; t += delta) {paintBrush.localPosition = new Vec3(lerp(lastUV.x, uv.x, t), lerp(lastUV.y, uv.y, t), zValue)paintBrush.drawExplicitly()}perform interpolation for every given parameter t ∈ [0, 1], t increasing every iteration for the given delta
lerp(x0, x1, t) function
return (1 - t) * x0 + t * x1linear interpolation between values x0 and x1, given parameter t
The finished script (also available as a part of the downloadable ZIP) can be attached to the Script component of the RootNode.
Step 9 - test the effect
The effect should now be finished and ready to have fun with!
The extended usage of the painting effect, with features such as erasing or changing the color and size of the paint brush, can be tested by using the Face Painting example applications for iOS (Objective-C) and Android (Java).
Download the ZIP with all the necessary assets and the final effect