Get your model ready for Perfect-Sync - 52 blendshapes for iPhone face Tracking -

日本語版

The following is a summary of information about the 52 blendshapes that are compatible with Apple ARKit.

- Detailed GIF animations for all 52 shapes.
- Real-world facial expressions to target for tracking
- Modeling tips and tricks

Click to enlarge

All 52 shapes are illustrated with GIF animations to show what we noticed and worked on during the prototyping process.

When viewed on a smart phone, the display is broken.
Please use the GoogleDocs file from the link below.

　　　　　↓↓↓

Description of 52 blendshapes for iPhone face Tracking - GoogleDocs

docs.google.com

Blendshapes LIST

	Blendshape Name
1	browInnerUp >>>ARKit documents			Raise the inside of the eyebrows Do not move the ends of the eyebrows Troubled Eyebrows	There are a total of 4 shape keys related to eyebrows  - browInnerUp  - browDownLeft / browDownRight  - browOuterUpLeft / browOuterUpRight  - noseSneerLeft / noseSneerRight There are a total of three shape keys for eye expressions  - eyeBlinkLeft / eyeBlinkRight  - eyeSquintLeft / eyeSquintRight  - eyeWideLeft / eyeWideRight The three eye shapes are not enough to cover the full range of expressions around the eyes. Including the eye movements in the eyebrow and cheek shape keys can enrich the variation. Be careful not to bury the eyebrows in the face mesh when moving multiple shapes of eyebrows at the same time. Maintain the position of the end of the eyebrow in browInnerUp. When browInnerUp and browOuterUpLeft/browOuterUpRight are combined at 100% each, the shape of a "naturally raised eyebrow (=surprised eyebrow)" is created. （browInnerUp+browOuterUp）
2	browDownLeft >>>ARKit documents			Lower the entire left eyebrow Quizzical eyebrows	Tilt and lower the entire left eyebrow. （browDown+browInnerUp） Push the upper eyelid down slightly. The movement of the upper eyelid should be such that when it overlaps with the blink of an eye, the eye is not buried and disappears.
3	browDownRight >>>ARKit documents			Lower the entire right eyebrow Quizzical eyebrows	Same as above
4	browOuterUpLeft >>>ARKit documents			Raise the outside of the left eyebrow Don't move the "eyebrow head" position.	In browOuterUp, maintain the position of the top of the brow Slightly lift the upper eyelid at the outer corner of the eye. When browInnerUp and browOuterUpLeft/browOuterUpRight are combined at 100% each, the shape of a "naturally raised eyebrow (=surprised eyebrow)" is created. （browInnerUp+browOuterUp）
5	browOuterUpRight >>>ARKit documents			Raise the outside of the right eyebrow Don't move the "eyebrow head" position.	Same as above
6	eyeLookUpLeft >>>ARKit documents			Turn the pupil of the left eye upward	Lower the center of the left upper eyelid Lower center of the lower eyelid Composite Surprised eye shape key so that the amount of change increases as the vertex moves toward the center. By focusing on the center of the eye, the eye moves smoothly when looking around the eyeball.
7	eyeLookUpRight >>>ARKit documents			Turn the pupil of the left eye downward	Lower the center of the right upper eyelid Lower center of the lower eyelid Same as above
8	eyeLookDownLeft >>>ARKit documents			Turn the pupil of the left eye downward	Lower the center of the left upper eyelid Lower center of the lower eyelid Composite Surprised eye shape key so that the amount of change increases as the vertex moves toward the center. By focusing on the center of the eye, the eye moves smoothly when looking around the eyeball.
9	eyeLookDownRight >>>ARKit documents			Turn the pupil of the left eye downward	Lower the center of the right upper eyelid Lower center of the lower eyelid Same as above
10	eyeLookInLeft >>>ARKit documents			Turn the pupil of the left eye inward	Look through the left eye socket. Move the eye vertices from the center to the top of the eye only. Composite the Surprised eye shape key so that the amount of change increases as the vertex moves toward the top of the eye. By adding a slope to the amount of change, the eye moves smoothly when looking around the eyeball.
11	eyeLookInRight >>>ARKit documents			Turn the pupil of the right  eye inward	Look through the right  eye socket. Move the eye vertices from the center to the top of the eye only. Same as above
12	eyeLookOutLeft >>>ARKit documents			Turn the pupil of the left eye outward	Look through the left corner of the eye. Move only the outer corner of the eye from the center of the vertex. Composite the Surprised eye shape key so that the amount of change increases as the vertex moves toward the outer corner of the eye. By adding a slope to the amount of change, the eye moves smoothly when looking around the eye.
13	eyeLookOutRight >>>ARKit documents			Turn the pupil of the right eye outward	Look through the right corner of the eye. Move only the outer corner of the eye from the center of the vertex. Same as above
14	eyeBlinkLeft >>>ARKit documents			Closing the left eye During tracking, the iPhone sensor detects the shielding of the pupil caused by the lowering of the eyelid as a downward rotation of the pupil. This may result in a bouncing motion of the pupil at the same time as blinking.	Normal eye closure If you set "Joy eye" closing instead of blinking here, you can easily close laughing eyes by face tracking (normal closing eyes will not be possible). In tracking animation, the pupils continue to rotate even when the eyes are about to close. For this reason, depending on the model, the pupil mesh may protrude when blinking. This can be prevented by adding a pupil scale to the closed eye shape.
15	eyeBlinkRight >>>ARKit documents			Closing the right eye During tracking, the iPhone sensor detects the shielding of the pupil caused by the lowering of the eyelid as a downward rotation of the pupil. This may result in a bouncing motion of the pupil at the same time as blinking.	Normal eye closure Same as above
16	eyeSquintLeft >>>ARKit documents			Smile with left eye	Smiling eyes with the left lower eyelid lifted. Note that the pupil is not closed in the ARKit reference, and is not the so-called "Joy eye" closed. The eyes are combined with the closed eyes, so if you simply set Joy Eyes Closed, it will fail. How to create "Joy Eyes Closed" using only face tracking 　(1) Minus 100% of the "closed eyes" shape. 　(2) Add 100% of the shape of "Joy eyes“　closed shape.  However, since the arc-shaped "Joy eyes closed" is a fiction, it tends to look unnatural in the middle when transformed from normal eyes. 　Face tracking does not work well with this middle expression, as it is the main focus. It is safer to use a key input that can be switched in an instant. Pushing up only the lower eyelid tends to change the impression of the face for characters with large eyes (making the face longer).
17	eyeSquintRight >>>ARKit documents			Smile with right  eye	Smiling eyes with the right lower eyelid lifted. Same as above
18	eyeWideLeft >>>ARKit documents			Widen the left eye	Open the left eye wide. For characters with large eyes, opening the eyelid may not be enough to see the whites of the eyes Shrink the pupil and adjust it so that the whites of the eyes are adequately visible.
19	eyeWideRight >>>ARKit documents			Widen the right eye	Open the right eye wide. Same as above
20	cheekPuff >>>ARKit documents			Cheeks puff out	Shape key to puffy cheeks You should puff them out so that they are visible from the front. Lips should be slightly forward. Pointy lips are cute. Be careful not to narrow them too much, or they will collapse when moved at the same time as the mouthPucker. The expression is easy to aim and activate. The face does not mix well with other shapes except the mouthPucker, so it is easy to create a posed face.
21	cheekSquintLeft >>>ARKit documents			Pull up the left cheek	Making a smile with the left cheek Movement as the cheek rises In cartoon-like models, moving parts such as the corners of the mouth and lower eyelids is less disruptive than moving the entire cheek. Stretching to the side makes the mouth expand when smiling. If the corners of the mouth are not stretched, the impression of the mouth can be maintained.
22	cheekSquintRight >>>ARKit documents			Pull up the right cheek	Making a smile with the right cheek Movement as the cheek rises Same as above
23	noseSneerLeft >>>ARKit documents			Frowning of the left nose and eyebrows Facial expression with eyebrows squeezed together in the center of the face	Frowning of the left eyebrow as an accompanying movement to the frowning of the nose The eyebrow is lowered and brought closer together. The entire eyebrow position is maintained. Be careful not to lower the eyebrow too much because it is combined with browDown. （browDown+noseSneer）
24	noseSneerRight >>>ARKit documents			Frowning of the right nose and eyebrows Facial expression with eyebrows squeezed together in the center of the face	Frowning of the right eyebrow as an accompanying movement to the frowning of the nose Same as above
25	jawOpen >>>ARKit documents			Open the mouth	Shape keys affect the impression of the character's face. Set a large mouth opening. When this opening is small, the mouth does not move much. This is paired with mouthClose, and when both are at 100%, the mouth closes with the jaw open. When both are at 100%, the mouth closes with the jaw open, and only the jaw moves to chew. （jawOpen+mouthClose） Basically, only jaw-related shapekeys move teeth. ・jawOpen ・jawForward ・jawLeft ・jawRight The shape that combines jawOpen and mouthUpperUp/mouthLowerDown is the maximum value of the vertical mouth opening. The shape that combines jawOpen with southDimple, southStretch, etc., respectively, is the maximum value of the mouth opening in the horizontal direction.
26	jawForward >>>ARKit documents			Move the jaw forward	The entire lower jaw moves forward Simultaneous movement of the mandible, tongue, and mandibular teeth
27	jawLeft >>>ARKit documents			Move the jaw left	The entire lower jaw moves to the left Simultaneous movement of the jaw, tongue, and mandibular teeth Because it is a movement that usually does not move much and is easy to control consciously, it is possible to intentionally activate a specific expression.
28	jawRight >>>ARKit documents			Move the jaw right	The entire lower jaw moves to the right Same as above
29	mouthFunnel >>>ARKit documents			Mouth when saying "Woo." Lips pouting and slightly open, showing closed teeth	The shape of the lips sticking out and spreading like a ladle Compose an expression that shows the teeth Do not move the teeth at all. *If you don't want to show teeth as much as possible as a character expression, add a movement to hide them in this shape key. Narrow the corners of the mouth to a sharp angle to give the impression of a mouth full of energy. Turn up the lips and thrust them out Lift the tip of the tongue slightly (because the tip of the tongue shrinks when people make this mouth shape) Adjust the mesh in the mouth so that the teeth do not fill in
30	mouthPucker >>>ARKit documents			Narrowing the width of the lips	Narrowing the width of the lips The narrower the lips, the fuller and more forward the lips will appear. Combine with mouthFunnel to create a kissing mouth. （mouthFunnel+mouthPucker） When combined with jawOpen, be careful not to make the narrowed mouth too long and narrow. Narrow the tongue at the same time as the lips (to prevent penetration of the corners of the mouth).
31	mouthLeft >>>ARKit documents			Bring the entire mouth to the left, not just the corners of the mouth.	Bring the entire mouth to the upper left. Do not move your teeth at all Rotate only the tongue to follow the mouth, allowing the tongue to move naturally （tongueOut+mouthLeft）
32	mouthRight >>>ARKit documents			Bring the entire mouth to the right, not just the corners of the mouth.	Bring the entire mouth to the upper right. Do not move your teeth at all Same as above
33	mouthRollUpper >>>ARKit documents			Pinch your upper lip between your teeth	The mouth itself disappears when the texture fills in, especially on toon-like models. Create a lip-biting shape while keeping the edge of the upper lip visible. Works in conjunction with mouthRollLower to create a squirming expression around the mouth. When actually moving, jawOpen is mixed in a little, so the movement is shallower than the created shape （mouthRollUpper+mouthRollLower）
34	mouthRollLower >>>ARKit documents			Pinch your lower lip between your teeth	Works in conjunction with mouthRollLower to create a squirming expression around the mouth. Same as above
35	mouthShrugUpper >>>ARKit documents			Lift your upper lip tightly	Shape that lifts the upper lip. Works in conjunction with mouthShrugLower to create a closed, frowning mouth. Also works with mouthPress to create a tightly closed mouth expression. （mouthShrug+mouthPress） The movement to pull down the corners of the mouth should be placed on either the upper or lower lip, as dividing it evenly between the upper and lower lips will cause the corners of the mouth to rattle.
36	mouthShrugLower >>>ARKit documents			Lift your lower lip tightly	Shape that lifts the lower lip. Works in combination with mouthShrugUpper to create a frowning, lopsided shape
37	mouthClose >>>ARKit documents			Close your mouth with your jaw open This shape does not work alone in tracking By opening and closing the jaw with the mouth closed, it becomes a chewing jaw movement. （jawOpen+mouthClose）	It looks like a broken shape, but it is not an abnormality in the model. Closing the mouth while keeping the jaw open Paired with jawOpen, when both are at 100%, the jaw is fully open, but the mouth is perfectly closed. From the jawOpen face, select the area around the mouth and gradually narrow the selection so that only the mouth is closed, resulting in the "jaw fully open and mouth perfectly closed" shape. From this state, we can create a mouthClose shape by further decreasing the jawOpen. The female model is not very pretty when her jaw is extended, so the jaw opening of the sample model Female has been made moderate. Move the tip of the tongue upward so that it does not penetrate the lower lip when you stick it out.
38	mouthSmileLeft >>>ARKit documents			Raise the left corner of the mouth	A smiling face with the left corner of the mouth pulled up Configure the shape of the mouth when smiling (e.g., make the corners of the mouth acute or obtuse). The mouth should look like a natural smile when combined with jawOpen. （jawOpen+mouthSmile） If you simply split the shape from the center to the left and right, the movement will be unnatural. Gradient the amount of change so that the left and right shapes add up to 100%. Lower the tongue slightly (because the corners of the mouth rise when people stick their tongues out strongly) （tongueOut+mouthSmile） Raising the corners of the mouth may cause the entire mouth to shift upward, changing the impression of the face as a character. Balance the mouth by lowering it appropriately to maintain its position. Do not move the teeth at all. *If you don't want to show the teeth as much as possible as a character expression, add a movement to hide the teeth in this shape key.
39	mouthSmileRight >>>ARKit documents			Raise the right corner of the mouth	A smiling face with the right corner of the mouth pulled up Do not move the teeth at all. *If you don't want to show the teeth as much as possible as a character expression, add a movement to hide the teeth in this shape key. Same as above
40	mouthFrownLeft >>>ARKit documents			Lower the left corner of the mouth Mainly works with the mouth closed	Lower the left corner of the mouth Mainly works with the mouth closed The shape of the mouth should be natural when merged with jawOpen. （jawOpen+mouthFrown）
41	mouthFrownRight >>>ARKit documents			Lower the right corner of the mouth Mainly works with the mouth closed	Lower the right corner of the mouth Mainly works with the mouth closed Same as above
42	mouthDimpleLeft >>>ARKit documents			Pull the left corner of the mouth backwards Open your mouth even wider to the side	Widen the left corner of the mouth horizontally In order not to change the shape of the corner of the mouth when composing with mouthSmile, etc., stretch the "middle part of the lips" instead of the corner of the mouth. （jawOpen+mouthDimple）
43	mouthDimpleRight >>>ARKit documents			Pull the right corner of the mouth backwards Open your mouth even wider to the side	Widen the right corner of the mouth horizontally Same as above
44	mouthUpperUpLeft >>>ARKit documents			Lift the left upper lip and bare the teeth Teeth bared in a mouthFunnel position without moving the jaw, teeth are most visible when the mouth is extended sideways （mouthFunnel+mouthUpperUp +mouthLowerDown +mouthDimple）	Lift the left upper lip. Do not move your teeth at all. Compared to the lower lip, the corner of the person's upper lip is raised more The entire upper lip will be shaped like a peanut. If you simply divide it into left and right from the center, it will look unnatural. Gradient the amount of change so that the left and right shapes add up to 100%. The shape of jawOpen and mouthUpperUp/mouthLowerDown is the maximum value of the vertical mouth opening.
45	mouthUpperUpRight >>>ARKit documents			Lift the right upper lip and bare the teeth	Lift the right upper lip. Do not move your teeth at all. Same as above
46	mouthLowerDownLeft >>>ARKit documents			Pull down the left lower lip and bare your teeth.	Pull down the left lower lip. Do not move your teeth at all. Compared to the upper lip, the middle part of a human's lower lip has a greater downward movement. If we simply split it from the center to the left and right, the movement will be unnatural. Gradient the amount of change so that the left and right shapes add up to 100%. The shape that combines jawOpen and mouthUpperUp/mouthLowerDown is the maximum value for the vertical opening of the mouth.
47	mouthLowerDownRight >>>ARKit documents			Pull down the right lower lip and bare your teeth.	Pull down the right lower lip. Do not move your teeth at all. Same as above
48	mouthPressLeft >>>ARKit documents			With your mouth closed, lift the left corner of your mouth tightly. Make a duck-like mouth.	Push up the left corner of your mouth and squash it. Works only when the mouth is squeezed, so it is easy to control consciously Can be used to add nuances such as a cat or duck mouth （mouthShrug+mouthPress 猫口バージョン） The cat's mouth must be symmetrical or it will look distorted. You can make it symmetrical by adding pre-prepared cat mouth shapes equally to the left and right mouthPresses.
49	mouthPressRight >>>ARKit documents			With your mouth closed, lift the right corner of your mouth tightly. Make a duck-like mouth.	Push up the right corner of your mouth and squash it. Same as above
50	mouthStretchLeft >>>ARKit documents			Pull the left corner of the mouth downward Compared to mouthFrown, the mouth moves more when it is open.	Pull down the left corner of the mouth to open the mouth further When you open your mouth wide, the corners of your mouth rise up, whereas the corners of your mouth are lowered to balance the movement. When the mouth is closed, it is difficult to create a beautiful movement by lowering the corners of the mouth. From a shape with 100% jawOpen, pull down the corners of the mouth to get the ideal shape, and then set jawOpen to -100% to get the proper movement. （jawOpen+mouthStretch）
51	mouthStretchRight >>>ARKit documents			Pull the right corner of the mouth downward Compared to mouthFrown, the mouth moves more when it is open.	Pull down the right corner of the mouth to open the mouth further Same as above
52	tongueOut >>>ARKit documents			Stick out the tongue	Stick the tongue straight out from the lips Deformation of the tongue to bend downward when the mouth is opened is done with jawOpen. Synthesize with jawOpen to adjust the width of the arch so that when the tongue is bent, the arch crosses the lower lip.

Sample Models

The latest version of the sample model that I created from the official version of VRoid VRM is available at the following Github. Please use the latest version in the release section.

github.com

The realistic face model is a sample model from FaceCap (borrowed with permission). It is included in the project which you can download here.

github.com

FaceCap is an iPhone face tracking application that has been used to create Youtube content for Idolish Seven. It does not support Perfect Sync, but if you prepare 52 BlendShapes that support Perfect Sync, FaceCap will also work. The face calibration function allows you to move the model clearly with a single tap, even if the person's expression does not move much.

Glossary of Terms

Apple ARKit：

A development kit available for iOS devices such as iPhone and iPad. One of the features provided is advanced face tracking. To take advantage of this, a 3D model with 52 corresponding blendshapes is required.

VRM：

A format for humanoid 3D models developed by the VRM Consortium. Users can freely add new expressions by composing existing expression blendshapes with Clip.

Perfect-Synk (a.k.a.)：

How to use face tracking with iPhone in combination with VRM specification. We add 52 Clips to the model and treat them as 52 different shapes that can be tracked in ARKit. Expression Clips can be adjusted by the user.

hinzka.hatenablog.com