DeepAR Studio

DeepAR Studio is a tool for AR asset creation within DeepAR SDK. The user can create various effects that are driven by face motion and expression. It imports .fbx models created by other 3D modeling tools (like Maya and Blender) and exports the prepared effects in the format that can be used by DeepAR SDK. Currently, DeepAR supports four types of effects (new effects are being added on a weekly basis):

  • Rigid objects
  • Deformable masks
  • Morph masks
  • Post processing effects

Rigid objects

Rigid objects are the basic type of effects that are supported by DeepAR SDK. It is a type of effect that is driven only by head position, meaning translation, rotation, and scale in 3D space. Typical effects in this category are glasses, hats, helmets and various other accessories.

Deformable masks

Deformable masks are the type of effects which, besides translation and rotation, are driven by facial expressions. Models are driven by the DeepAR reference head model in two ways – via vertices and bones. For more information see the Reference mask model section.

Morph masks

Morph mask effects are effects that utilize blend shape as geometry deformators. When you want to create a mask and use the reference model vertices to drive its deformation, the easiest way is to make a blend shape for the reference head model. That way the vertices/bones that drive the resulting model will be preserved. As a side effect to this, the transformation effect can be implemented – you can change the blend shape weight over time and user will see itself transformed into the desired effect/mask.

Post processing effects

Post processing effects are applied over the entire frame (including all the objects rendered). Effects in this category use the Postprocessing Layer and specific postprocessing shaders, such as the LUT Fixed Intensity shader.

Reference mask model

Reference mask model is the underlying model that is driven by DeepAR face tracking technology. Users of the SDK use it as a starting point in creating their own effects in 3D modeling tools (like Blender or Maya).

Geometry

It’s geometry (vertices) and deformators (bones and blend shapes) are used to drive the user created masks. Below is the image of the reference model imported in Blender.

Reference model imported in Blender Reference model vertex color values

The reference model has a geometry that consists of 730 vertices, all of which are driven by the underlying face tracking technology. The engine differentiates every vertex by its unique color property and for that reason, it shouldn’t be changed – otherwise, it will not be able to track a given vertex.

Bones

Other than vertices, user created masks can be driven by bones. Bones are used when the model isn’t created from the reference model. In that case it is easier to map significant points of interest in the reference to the resulting model since the number of bones is smaller than number of vertices. The user doesn’t have to define every bone in the resulting model. Bones are defined by their names which is in the format of “jxx” where the “xx” part is the number of the bone in the reference model. Below is the map of bones available in reference model:

Reference model bones

Other

Models that can be read by the DeepAR Studio can are arbitrary .fbx valid models with any number of gemoetries organized in an FBX nodes hierarchy. Other than geometries and bones, the DeepAR Studio loads fbx material data for each mesh/geometry. Material data can be additionally edited and refined in the DeepAR Studio, but it is important to add the placeholder material in the .fbx model itself. For the vertices of a mesh, the Studio reads and tracks following data:

  • Position
  • Color
  • UV
  • Normal
  • Tangent

All these parameters can be used in shader computations.

UI explained

The DeepAR Studio uses a camera to provide user with real time feedback how the effect will look in the mobile app. Every parameter change will be reflected immediately in the live preview window. For usability purposes some parameters will require clicking on the “Refresh model” button for changes to be reflected.

DeepAR Studio UI with loaded effect

Window toolbar

Besides the standard OSX options like minimizing and maximizing the window, the main window toolbar functions include loading .fbx files and previously generated DeepAR binary models in the scene. Additionaly, when users are done editing the effect, the toolbar provides the option to export the effect to the DeepAR binary format – File->Export Model

DeepAR Studio window toolbar

Node hierarchy tree

The node hierarchy panel provides a hierarchical view of loaded model nodes. To edit certain properties of the nodes, and meshes inside those nodes, the user needs to select the given node in the hierarchy. A node doesn’t need to have a visual representation in the scene, it can be a container for other nodes and scene objects. Usually the leaf node elements are concrete objects like geometries/meshes which are visible in the scene, or bones objects defined for the model.

DeepAR Studio node hierarchy

Node properties

In this panel the user can edit certain properties that are important for nodes in the hieararchy. Those include:

  • Layer
  • Morph mode
  • Position mode
DeepAR Studio node properties

Layer

Nodes in the virtual scene can be placed in layers which allow the user to adjust the rendering order for each node. The higher the layer value for node is, the later the rendering of given node will occur. This provides the user with the basic z-ordering functionality for node objects. This is comparable to the layer objects in Photoshop or Gimp. Some layers such as the Postprocessing layer have special behaviours and should only be used for their intended purpouse.

Morph mode

Morph mode is used to enable or disable morphing. Morphing is used to deform the face using a blend shape. The possible morph modes are:

  • Morph disabled
  • Morph enabled: if morphing is enabled, the user can use Camera Texture as a texture for the materials.
  • Morph enabled (with alpha): morphing is enabled, but alpha mask can be used to specify what part of the mesh is morphed.

Position mode

Position mode determines where the object will be placed. This property should be set only on the root node. There are following options:

  • Position in face space: the object will track the face – it will match the translation and rotation of the face
  • Position in camera space: object will NOT track the face – it will have a fixed position, rotation and scale relative to the camera.

Transform toolbox

Transform toolbox panel allows a user to position the nodes in 3D space via position, rotation and scale parameters in the x, y and z axis. Transform parameters can be set on parent nodes or on leaf/child nodes. Transformation parameters on parent nodes affect all the child nodes down the hiearchy.

DeepAR Studio transform toolbox

Texture list

Texture list panel allows user to load any textures that will be applied to model or parts of the model. DeepAR Studio, by itself, doesn’t load any textures that are found in a .fbx model (stored internally or externally); that means that every texture must be loaded in the DeepAR Studio by clicking the “Add texture” button. A loaded texture can be deleted by hitting the “Delete” button in the texture list table. The DeepAR Studio provides a default camera texture and a post processing texture for each model. The camera texture can be applied to models and the content of that texture corresponds to the current camera frame, the post processing texture should be applied to postprocessing effects.

Texture type

The texture can have the following types:

  • Regular: texture used for diffuse color
  • Animation: animated image loop sequence – can be triggered by mouth open/close (coming soon)
  • Cube map: texture that can be used for realistic reflections (coming soon)
DeepAR Studio texture list

Materials editor

The Materials pane contains the list of all materials that are imported from the .fbx file. A user can select the node in the Node hierarchy pane and if the selected node has a mesh associated with the material, the material will be selected in the material list. When selected, the material details will be shown below.

States

The materials states can control the render behaviour of the material.

  • RGB write: renderer will write to RGB color buffer
  • Alpha write: renderer will write to Alpha buffer.
  • Depth write: renderer will write to depth buffer.
  • Alpha blend: alpha blending enabled/disabled
  • Cull clockwise: cull the triangles that are specified in clockwise order.

Shader editor

For each material, the user needs to select the shader. This can be done in the shader combo box. Currently available shaders are:

  • Unlit texture: only render texture without any lighting
  • Unlit color: only render solid color without any lighting
  • Unlit texture color: render texture multiplied with color without any lighting
  • Color lighting: render color using lighting model. The user can specify diffuse color, specular color and ambient color. The alpha values of the colors are used to specify the amount of specular or ambient color.
  • Texture color lighting: same as the above, but uses diffuse texture
  • Skin blend: more complex shader that can be used to blend texture with underlying skin.
  • Simple morph: a shader used with “Morph mode (with alpha)”.
  • Simple Morph Tex: same as the above, but uses diffuse texture.
  • LUT Fixed Intensity: A shader used with Postprocessing effects to apply custom color transformations to the frame.

Blend shape editor

If the selected node has mesh that has blend shapes, they will be listed in the Blend shapes pane. The user can change blend shape value from -100 to 100. The blend shapes are mostly used for face deformation with Morph mode enabled.

Post processing effects

The user can also create post processing video effects. Currently the only avaliable post processing effect is the LUT filter.

Examples

Following are examples for 4 different kind of assets that can be currently made with the DeepAR Studio (though more effects are coming soon): rigid, deformable, morphing and postprocessing effects. All the .fbx models and textures used in the examples are included with this documentation.

Aviators

This example will show how to prepare a rigid model effect for the DeepAR Studio – we’ll be using an aviators glasses model.

1. Load .fbx model

The first step is to load the aviators_v6.fbx model in the scene. File->Import fbx->choose aviators_v6.fbx in the finder explorer. This is the result:

Aviators step 1
2. Setup face texture

As you can see from the previous screenshot, we have two group of geometries in the scene, one is the aviators model and the second one is the face model. The face model acts as an occlusion mask for glasses, i.e. when user turns his head certain parts of aviator glasses need to be occluded. However, the mask must be transparent and to achieve that we will need to apply camera texture to face model part:

  • Select “Mesh” node in node hieararchy
  • The Studio selects the appropriate material for this mesh which, in this case, is “lambert2”
  • In the material details, “Unlit texture” is selected as shader
  • We need to set the texture uniform “s_texColor” to “Camera texture” and click “Refresh model”. At this point nothing has changed yet even though the camera texture is set to mesh. That is beacause the camera texture doesn’t have appropriate “Morph mode” to accomplish the desired effect
  • In Node properties->Morph mode->select option 1 from the drop down box. This option tells the SDK that only part of the Camera texture needs to be applied to mesh model and not the whole camera texture. The part that gets applied is dynamically calculated in every frame by projecting the face model vertices onto screen and taking that part of texture.
Aviators step 2

Now the face is set up, let’s move to next step.

3. Load aviators texture

We need to load the texture which will be applied to aviator glasses model. In texture list panel hit Add texture -> choose aviators_v3.png

4. Apply textures to materials

Once we have loaded the texture, we need to apply it to materials. If we open the texture, we will see that all parts of aviators are added to single texture. Therefore for every material in the scene we set the shader to “Unlit shader” and s_texColor uniform to aviators_v3.png texture, except for the material that is responsible for lenses. For lenses material (“lambert1”) we will a use different shader because the lenses need to be colored and semi-transparent, and for that puprose we will use “Color lighting” shader. For this example we’ve used the following parameters for shader uniforms:

  • u_diffuseColor: R: 0, G: 0, B:0, opacitiy: 35%
  • u_specularColor: R: 255, G:255, B:255, opacity: 25%
  • u_ambientColor: R:146, G:0, B:12, opacitiy: 100%

Other than shader uniforms, we need to set parameter “Cull clockwise” for “lambert1” parameter to enabled – meaning the Cull clockwise checkbox needs to be checked. This is the final result:

Aviators step 4

Lion

The lion mask is an example of how to use a mesh that is driven by bones. You can find lion.fbx and lion.png in the examples folder. The lion.fbx contains skinned mesh using the bones/joints that are specifically named so the DeepAR engine can make the connection between the specific face point and the bone. Here are the steps to create lion mask:

  • Open DeepAR Studio, go to File, Import FBX, choose lion.fbx and click open
  • The DeepAR Studio will import the lion mesh, materials and show the non-textured material in the central view.
  • To load the texture, click “Add texture” on the right hand side, choose lion.png and click open
  • The next step is to assign the texture to the material. Select the only material (lambert6) in the material list. We’ll keep the preselected shader (Unlit Texture) and change s_texColor from Not selected to lion.png.
  • We should immediately see lion mask on the face. Since lion.fbx already has correctly skinned mesh with named bones, everything works out of the box.
  • After we’re happy with the result, we can go to File, Export model and Save. The exported file can be loaded in the mobile SDK and added to the scene. The exported file can also be opened in the DeepAR Studio and edited further.

Bigmouth

The bigmouth is an example of how to use a mesh that is driven by vertices. Each vertex of this mesh is directly driven by face tracking. This kind of approach is useful when user wants effects like face deformation, face paint or makeup. For this approach, the user needs to use original_face.fbx (available in examples folder) as a starting point. The example workflow for face deformation is the following:

  • Open original_face.fbx in 3D editor (e.g. Maya)
  • Create a blend shape that deforms the original face (e.g. make the mouth bigger, eyes smaller)
  • Export the new fbx
  • The user should not add any new vertices and should not change the color of the vertices. The number of vertices in the exported fbx needs to be 730.

Here’s how the mesh created by this workflow can be used in the DeepAR Studio. We’ll use bigmouth.fbx which is available in the examples folder.

  • Open the DeepAR Studio, go to File, Import FBX, choose bigmouth.fbx and click Open
  • Select lambert3 material and set Camera Texture to s_texColor.
  • You should immediately see deformed face. The DeepAR Studio can recognize what is the type of the fbx loaded and sets the initial settings accordingly.
  • You can change the amount of deformation by selecting the node that has mesh that has blend shape (in this case the node “Mesh”) and changing the blend shape value. The blend shape is visible in the lower right corner of the DeepAR Studio. You should see a blend shape called bigMouth_BS. After changing the value, you need to click on Refresh model to apply the changes.
  • When we’re happy with the results, we can export the effect and use it in the mobile SDK.