API documentation

Detailed docs here.

DeepAR SDK functionality is exposed through the class called DeepAR. Depending on your use case, the SDK can be used in several different modes:

  • By rendering type: on-screen and off-screen. On-screen rendering mode is used when we want to display the result of DeepAR SDK on the screen to the user in real-time, like rendering camera stream and displaying it somewhere in the app UI. Off-screen rendering is used if we want to process image data but not necessarily show the result right away and the real-time processing is not that important like processing a pre-recorded video.

  • Live mode on/off - if we need to process the result and display them in real-time we can turn the live mode on which will optimize the inner workings of the engine for performance. Live mode off is used if we do not need continuous real-time image processing, e.g. processing a single image. With the live mode off the engine is optimized for preserving processing and memory resources.

  • Computer vision only mode - with this mode on the engine will not do any rendering whatsoever it will only output computer vision data like detected faces positions, rotations, emotion estimation, etc.

Using adequate API methods the user should be able to transition to any mode they need without the need to reinitialize the engine.

DeepAR

void initialize(Context context, AREventListener eventListener)

Initializes the DeepAR engine. DeepAR engine is contained in the class DeepAR. This class should be instantiated and the method initialize should be called before use. This is typically done in onCreate method in Android Activity. Any method on

void initialize(Context context, AREventListener eventListener, String modelPath)

Initializes the DeepAR engine. The modelPath parameter can be used if the model file is downloaded over the air. The model file is usually located in the assets folder. The default value for this parameter is file:///android_asset/models.bin and this value is used if modelPath is null. Other parameters are the same as above.

void release()

Releases all resources required by DeepAR. Must be called after the rendering surface is destroyed. It's usually called in onDestroy method in Android Activity.


@Override
protected void onDestroy() {
super.onDestroy();
deepAR.release();
}

void receiveFrame(ByteBuffer data, int width, int height, int orientation, boolean mirror, DeepARImageFormat imageFormat, int pixelStride)

DeepAR can be used with Camera, Camera2 or CameraX API. The parameter data should be image byte data in YUV_NV21 format (Camera API) or YUV_420_888 format (Camera2 and CameraX API). ByteBuffer must be allocated using allocateDirect. When using CameraX API and you get 3 planes of buffers, just put all the planes in one buffer in the Y, V, U order. The orientation parameter is the camera sensor orientation, which can be read from CameraInfo object. ImageFormat parameter tells whether you are passing a YUV_NV21 frame or YUV_422_888. PixelStride is always 1 for Camera API, for Camera2 API pass the pixel stride of U or V plane.

void switchEffect(String slot, String path)

The method used to switch any effect in the scene. Effects are places in slots. Every slot is identified by its unique name and can hold one effect at any given moment. Every subsequent call to this method removes the effect that was previously displayed in this slot. To clear effect from a slot, call this method with null for path param. Using this method user can combine multiple effects in the scene, like using a face mask and a filter effect: When tracking multiple faces is enabled this method only affects the first tracked face.


deepAR.switchEffect("mask","file:///android_asset/flowers");
deepAR.switchEffect("filters","file:///android_asset/tv80");

Slot parameters are arbitrary strings. The path parameter is a string path to an effect file in the project which has been exported from DeepAR Studio.

void switchEffect(String slot, String path, int face)

This method is used to switch the effect, similar to the previous method. The face parameter is used when assigning the effect to a particular face. Different faces should use different slots such as "mask_f0" and "mask_f1" instead of "mask" in the previous example. DeepAR SDK supports tracking 4 faces in parallel (by default, SDK supports only one face - if you need more, please contact the support).


// apply flowers effect to the first face
deepAR.switchEffect("mask_f0","file:///android_asset/flowers",0);
// apply beard effect to the second face
deepAR.switchEffect("mask_f1","file:///android_asset/beard",1);
// replace the effect on the first face with the lion
deepAR.switchEffect("mask_f0","file:///android_asset/lion",0);
// remove the beard effect from the second face
deepAR.switchEffect("mask_f1",null,1);

void switchEffect(String slot, InputStream inputStream)

Loads effect from the InputStream.

void setFaceDetectionSensitivity(int sensitivity)

This method allows the user to change face detection sensitivity. The sensitivity parameter can range from 0 to 3, where 0 is the fastest but might not recognize smaller (further away) faces, and 3 is the slowest but will find smaller faces. By default, this parameter is set to 1.

void setAREventListener(AREventListener listener)

Sets the AREventListener for DeepAR object.

void setRenderSurface(Surface surface, int width, int height)

Calling this method sets the engine in On-screen rendering mode (cancels off-screen rendering if set before this call).

Passes the render surface to the DeepAR engine. The Surface is typically obtained in onSurfaceChanged method in SurfaceHolder.Callback. DeepAR engine uses SurfaceView to render 3D augmented reality elements. The SDK user is required to place the SurfaceView in the view hierarchy (through XML layout or manually from code). The user also needs to add a surface holder callback, for example: surfaceView.getHolder().addCallback(this);.

Here's an example of how to use SurfaceHolder.Callback with DeepAR


private DeepAR deepAR;
...

@Override
public void surfaceCreated(SurfaceHolder surfaceHolder) {
}

@Override
public void surfaceChanged(SurfaceHolder surfaceHolder, int format, int width, int height) {
// Pass the surface object along with width and height to DeepAR instance. DeepAR object must be instantiated and initialized at this point.
deepAR.setRenderSurface(surfaceHolder.getSurface(), width, height);
}

@Override
public void surfaceDestroyed(SurfaceHolder surfaceHolder) {
// If the surface is destroyed, DeepAR engine must be notified by passing null as the Surface object. The rendering will be paused after this call.
deepAR.setRenderSurface(null,0,0);
}

...

}

void setOffscreenRendering(final int width, final int height, DeepARPixelFormat format)

Calling this method sets the engine in Off-screen rendering mode. The engine will render the results of engine processing in the off-screen buffer which will be available to the user in the frameAvailable method of the AREventListener. Calling this method will automatically stop any on-screen rendering.

width and height define the output size of the off-screen rendered image.

format - specifies the output Image frame pixel format of frameAvailable method. Sometimes 3rd party libraries that do additional frame processing after DeepAR expect the input frames in a specific format (like video streaming libs) and it is controlled by this parameter.

void startCapture(DeepARPixelFormat format)

Starts capturing the DeepAR processing output in an off-screen buffer. AREventListener method frameAvailable will be called with the contents of that buffer any time a new frame is ready meaning the frame is processed by the DeepAR engine. frameAvailable will not be called unless this method is called explicitly. User should call this method if they need to further process the DeepAR output like sending them over the network to implement video chat functionality. This method allows the DeepAR to be simultaneously in both on-screen and off-screen rendering mode - one could display the frames in UI and collect them in the frameAvailable.

format - specifies the output Image frame pixel format of frameAvailable method. Sometimes 3rd party libraries that do additional frame processing after DeepAR expect the input frames in a specific format (like video streaming libs) and it is controlled by this parameter.

void startCapture(int width, int height, float xmin, float xmax, float ymin, float ymax, DeepARPixelFormat format)

Overload on the startCapture method with added parameters to specify the subframe of the currently rendered content. startCapture() internally calls this method with the following parameters:

startCapture(renderWidth, renderHeight, 0, 1, 0, 1);

width and height control the output resolution and, xmax, ymin and ymax control the sub rectangle of the processed image in the unit coordinates.

format - specifies the output Image frame pixel format of frameAvailable method. Sometimes 3rd party libraries that do additional frame processing after DeepAR expect the input frames in a specific format (like video streaming libs) and it is controlled by this parameter.

void stopCapture()

Stops capturing the content of the DeepAR rendering process in the off-screen buffer. Should be called if the user does not need post-processing features to preserve memory and computational resources.

void setVisionOnly()

Set DeepAR in the vision-only mode. In this mode, DeepAR stops all rendering and it only outputs computer vision data which consists of tracking users FaceData. See Getting the face coordinates section below.

To exit vision only mode call either setRenderingSurface or setOffscreenRendering to switch DeepAR to rendering mode.

void changeLiveMode(boolean value)

This is an optimization method and it allows the user to indicate the DeepAR in which mode it should operate. If called with true value, DeepAR will expect a continuous flow of new frames and it will optimize its inner processes for such workload. An example of this is the typical use case of processing the frames from the camera stream.

If called with false it will optimize for preserving resources and memory by pausing the rendering after each processed frame. A typical use case for this is when the user needs to process just one image. In that case, the user will feed the image to DeepAR by calling receiveFrame method, and DeepAR would process it and stop rendering until a new frame is received. If we did so when the DeepAR is in live mode, it would process the same frame over and over again without ever stopping the rendering process.

void setPaused(boolean paused)

Pauses/unpauses the rendering. This method will not release any resources and should be used only for temporary pause (e.g. user goes to the next screen). Use the shutdown method to stop the engine and release the resources.

void takeScreenshot()

Captures a screenshot of the current screen. When a screenshot is done AREventListener method screenshotTaken will be called with a Bitmap parameter which will be the resulting screenshot.

void startVideoRecording(String filePath)

Starts video recording of the camera preview. The video will be the same size as SurfaceView. Video recording events can be observed through AREventListener implementation.

void startVideoRecording(String filePath, int outputWidth, int outputHeight)

Starts video recording, the outputWidth, and outputHeight parameters can be used to specify output video resolution. The SurfaceView will be skewed to the desired resolution. To preserve the aspect ratio, enter the desired output width or height and set the other parameter to zero.


// Record video with width 300 and calculate height to preserve aspect ratio
deepAR.startVideoRecording(Environment.getExternalStorageDirectory().toString() + File.separator + "video.mp4", 300, 0);

void startVideoRecording(String filePath, Rect subframe, int outputWidth, int outputHeight)

Starts video recording. The subframe parameter can be used to record only part of SurfaceView. The subframe Rect should be given in pixels


// Record the left half of the screen
Rect subframe = new Rect();
subframe.left = 0;
subframe.right = deepAR.getRenderWidth();
subframe.top = 0;
subframe.bottom = deepAR.getRenderHeight();

//width 300, height to be calculated
deepAR.startVideoRecording(Environment.getExternalStorageDirectory().toString() + File.separator + "video.mp4", subframe, deepAR.getRenderWidth()/2, deepAR.getRenderHeight());

void startVideoRecording(String filePath, Rect subframe, int outputWidth, int outputHeight, boolean recordAudio)

Starts video recording. This method allows the audio recording to be enabled or disabled for a recording session. When using variants of this method without the recordAudio parameter, audio recording will be on by default. Other parameters work the same as the method above.

void startVideoRecording()

If the previous method was called with prepare=true, this method can be used to actually start the video recording.

void stopVideoRecording()

Stops video recording and starts the process of saving the recorded video to the file system. When the file is saved, the method videoRecordingFinished will be called.

void setAudioMute(boolean mute)

Mutes/unmutes the audio while video recording.

int getBitRate() / void setBitRate(int bitRate)

Gets or sets the video recording bit rate. The default is 0.15 * frame rate * width * height

int getiFrameInterval() / void setiFrameInterval(int iFrameInterval)

Gets or sets the I frame interval (MediaFormat.KEY_I_FRAME_INTERVAL) for video recording. The default is 5.

int getKeyFrameRate() / void setKeyFrameRate(int keyFrameRate)

Gets or sets the key frame rate (MediaFormat.KEY_FRAME_RATE). The default is 25.

void setAntialiasingLevel(int level)

Sets the MSAA level. The allowed levels are 0 (MSAA off), 2, 4, 8, 16. This should be called before initialize to have any effect.

void changeParameterFloat(String gameObject, String component, String parameter, float val)

This method allows the user to change float parameters during runtime. The method requires a Game Object name, Component name, Parameter name, and a float value.

  • gameObject is the name of the Game Object in the hierarchy as visible in the Studio. The name should be unique for the method to work properly.

  • component is the internal component type.

  • parameter is the name or the parameter to be changed, for example, the name of the blendshape on a mesh. The parameter must exist for the component.

  • floatValue is the new value to be applied to the parameter.


// Change the weight of the blendShape1.cuteBS blendshape on the Mesh Game Object.
deepAR.changeParameterFloat("Mesh", "MeshRenderer", "blendShape1.cuteBS", 5.0f);

void changeParameterVec4(String gameObject, String component, String parameter, float valX, float valY, float valZ, float valW)

Same as above except the method takes four float values which represent one Vector4. Can be used to change material color. If multiple Game Objects share the same material, changing the parameter once for any Game Object using the material will change it for all Game Objects. To change a uniform on a material, such as color, the parameter must use the internal uniform name. These names can be found in the shader files.


// Change the color of the u_diffuseColor uniform (diffuse color for Standard shader) on the material applied to the Mesh Game Object to a semi-transparent green.
deepAR.changeParameterVec4("Mesh", "DynamicMeshRenderer", "u_diffuseColor", 0.0f, 1.0f, 0.0f, 0.25f);

void changeParameterVec3(String gameObject, String component, String parameter, float valX, float valY, float valZ)

Same as above except the method takes three float values which represent one Vector3. Can be used to change the transform of a Game Object at runtime. Rotations should be written as Euler angles in degrees. To set a parameter on GameObject, the component parameter should be an empty string.


// Change the position, rotation and scale of the RootNode game object
deepAR.changeParameterVec3("RootNode", "", "position", 0.06f, -0.02f, 0f);
deepAR.changeParameterVec3("RootNode", "", "rotation", 0f, -30f, 0f);
deepAR.changeParameterVec3("RootNode", "", "scale", 2f, 0.5f, 1f);

void changeParameterBool(String gameObject, String component, String parameter, boolean val)

Same as above except val takes a bool value. Can be used to disable or enable GameObjects at runtime. To set a parameter on GameObject, the component parameter should be an empty string..


// Disable the Mesh game object
deepAR.changeParameterBool("Mesh", "", "enabled", false);

void changeParameterTexture(String gameObject, String component, String parameter, Bitmap image)

This method allows the user to change the texture at runtime. The component should be "MeshRenderer". The parameter is the name of the texture that should be changed, this name can be found in the shader that is applied to the material/GameObject.


// Load texture
Bitmap bitmap = BitmapFactory.decodeResource(getResources(), R.drawable.texture);

// Change the texture of the material attached to Quad game object. s_texColor is the name of the texture uniform in the shader code.
deepAR.changeParameterTexture("Quad", "MeshRenderer", "s_texColor", bitmap);

void moveGameObject(String selectedGameObject, String targetGameObject)

Moves the selected game object from its current position in a tree and sets it as a direct child of a target game object. This is equivalent to moving around a node in the node hierarchy in the DeepAR Studio.

void fireTrigger(String trigger)

This method allows the user to fire a custom animation trigger for model animations from code. To fire a custom trigger, the trigger string must match the custom trigger set in the Studio when creating the effect.

void touchStart()

Tells the DeepAR that the touch started for the Hide on touch component of effects.

void touchEnd()

Tells the DeepAR that the touch ended for the Hide on touch component of effects.

void setLicenseKey(String key)

The method used to set the license key for your project. The license key is linked to the applicationId it's created for. Tracking will not work unless a matching license key and applicationId are used.

FaceTrackedCallback getFaceTrackedCallback() / void setFaceTrackedCallback(FaceTrackedCallback faceTrackedCallback)

Gets or sets the callback function for handling face coordinate data for a single tracked face, see Getting the face coordinates

DeepARPixelFormat

Used to specify the pixel output format of the Image frame of frameAvailable method.

public enum DeepARPixelFormat {
RGBA_8888,
ARGB_8888,
BGRA_8888,
ABGR_8888;
}


DeepARImageFormat

Used to specify the image format for the receieveFrame method.

public enum DeepARImageFormat {
YUV_NV21,
YUV_420_888;
}

AREventListener

Listener for asynchronous actions returned from the DeepAR class like receiving screenshot results etc. All listener methods are called on the main thread.

void initialized()

Called when the DeepAR is initialized. DeepAR methods should not be called before the initialization is completed.

public void frameAvailable(Image frame)

Each processed frame will be available here as an Image object. Make sure to call startCapture on DeepAR object otherwise this method will not be called.

void screenshotTaken(Bitmap screenshot)

Called when the screen capture is finished. The result is given as an Android Bitmap object.

void faceVisibilityChanged(boolean faceVisible)

Called when the user's face becomes visible or invisible. This event can be used to show a hint to the user to position the face in the camera field of view.

void videoRecordingStarted()

The start of the video recording process is not synchronous, so this method will be called when the video recording is started.

void videoRecordingFinished()

Called when the video recording is finshed.

void shutdownFinished()

Called when engine shutdown is finished after calling release().

void videoRecordingFailed()

Called if the video recording fals.

void error(String error)

Called when an error occurs - like the model path not found or the effect file failed to load.

void effectSwitched(String slot)

Called when an effect has been switched on a slot given by the slot parameter.

void imageVisibilityChanged(String gameObjectName, boolean imageVisible)

Called when a natural image is being tracked and the visibility has changed. The gameObjectName contains the name of the game object with the Natural image position component attached and the current visibility is given by the imageVisible parameter.


Getting the face coordinates

The user can get the raw face coordinates if needed. The coordinates are packed in the FaceData class


public static class FaceData {

// True if the face is visible in the current frame, false otherwise
public boolean faceDetected;

// Translation and rotation (euler angles in degrees) of the face center in 3D coordinates (x,y,z), calculated based on the vertical field of view of 50 degrees
public float[] translation;
public float[] rotation;

// The pose matrix 4x4 containing the face translation and rotation.
public float[] poseMatrix;

// 68 face landmarks in 3D space, relative to the face center (x1,y1,z1,x2,y2,z2,...)
public float[] landmarks;

// 68 face landmarks in image space (pixels) (x1,y1,x2,y2,...)
public float[] landmarks2d;

// Rectangle covering the face in viewport coordinates, normalized to the [-1, 1] range (x, y, width, height)
public float[] faceRect;

...

To get the FaceData instance, the user needs to call setFaceTrackedCallback. Here's an example


deepAR.setFaceTrackedCallback(new DeepAR.FaceTrackedCallback() {
@Override
public void faceTracked(DeepAR.FaceData faceData) {

}
});

To stop getting the face data, simply call setFaceTrackedCallback with a null parameter.

The FaceTrackedCallback.faceTracked will be called on each rendered frame. The user can get the face data even if the renderer is not running (e.g. the surface was never passed to the DeepAR instance). In that case, the face tracking needs to be set to work in computer vision only mode. Check the setVisionOnly() method definition above:


//...
deepAR.initialize(this, this);
deepAR.setVisionOnly();
deepAR.setFaceTrackedCallback(new DeepAR.FaceTrackedCallback() {
@Override
public void faceTracked(DeepAR.FaceData faceData) {

}
});


Tracking multiple faces

When working with multiple faces use the void switchEffect(String slot, String path, int face) method to apply face tracking effects such as masks to the desired face. The void switchEffect(String slot, String path) method applies the effect to the first face (index 0) and should not be used for applying face tracking effects, but for code readability, it is still recommended for camera space effects such as filters there should only be one of. Different faces should use different slots such as "mask_f0" and "mask_f1" instead of "mask" to avoid possible conflicts.

Correct usage:


// apply flowers effect to the first face
deepAR.switchEffect("mask_f0","file:///android_asset/flowers",0);
// apply tv80 filter, internally it is assigned to the first face
deepAR.switchEffect("filters","file:///android_asset/tv80");
// apply beard effect to the second face
deepAR.switchEffect("mask_f1","file:///android_asset/beard",1);
// replace the effect on the first face with the lion
deepAR.switchEffect("mask_f0","file:///android_asset/lion",0);
// remove the beard effect from the second face
deepAR.switchEffect("mask_f1",null,1);
// remove the lion effect from the first face
deepAR.switchEffect("mask_f0",null,0);
// remove the tv80 filter
deepAR.switchEffect("filters", null);

Not recommended :


// apply flowers effect to the first face// not clear since the face was not explicitly given
deepAR.switchEffect("mask_f0","file:///android_asset/flowers");
// apply tv80 filter, explicitlly assigning it to the first face
//reduces code readability since filter effects affect the whole screen
deepAR.switchEffect("filters","file:///android_asset/tv80", 0);

Did this answer your question?