Studio Glove - Unity Streaming from Hand Engine via SDK

Unity Streaming

NOTE:
- Ensure all gloves have been paired and calibrated in Hand Engine.
- Make sure to use either 2021 or 2022 Long-Term Support (LTS) versions of Unity.
- Make sure your project is set to a 3D template.

1. Download the most up-to-date plugin from the Downloads section of your StretchSense Account.
2. Extract the downloaded .zip file and copy the .tgz file into the Packages folder in your new or existing project directory.

NOTE: If you are working on a new project you will have to launch it first so the project directory is created, then close Unity in order to move the .tgz file into the Packages folder.

3. Launch your Unity project, go to Window -> Package Manager
   
4. In the Package Manager window, click on the '+' icon and select 'Add package from tarball...'. Then go into the Packages folder from your project and select the .tgz file.
   
5. The Unity plugin has been successfully set up.
6. Follow the steps from the next session to test the sample files StretchSense provides.

Setting Up the Hand Engine Sample Files
 

To experience the functionality of the plugin using the sample files provided by StretchSense, you have the option to import them in the Package Manager window.

1. Select the 'StretchSense Hand Engine Studio Plugin for Unity'  from the list and click on 'Samples'. Then click on 'Import'
   

NOTE: 
If you are using a 3D (URP) template for your project, it is necessary to convert all materials from your Assets folder to URP after importing the sample files. Otherwise, the scene will display assets in a bright pink color.

1. Launch your project, and go to Window -> Rendering -> Render Pipeline Converter
   

2. Select 'Material Upgrade' and click on 'Initialize And Convert'. Click on 'Save and Continue'. 
   

Back into your main project, go to the Project tab and navigate to:

Assets > Samples > StretchSense Hand Engine Studio Plugin Plugin for Unity > 3.0.1 > Hand Engine Studio PLugin for Unity > Prefabs

Drag and drop both the 'Example Environment' and 'StretchSense Hand Rig'  prefab files into the Hierarchy tab on a space.

2. Double-click on the StretchSense Hand Rig to zoom into the hands and press Play at the top to be able to see the fingers moving.
   

TIP: 
If you find that the Game view camera is positioned too far from the hands, simply click on the Scene tab to get a closer view of the hands.

TIP:
In the Hierarchy tab If you expand the StretchSense Hand Rig prefab and click on 'HandEngineClient'  you will be able to see what information is being received from Hand Engine in the Inspector tab.

 
Custom Character Retargeting in Unity

The following instructions require you to have already imported the StretchSense Studio Plugin for Unity into your Unity project. See Quick Start instructions above for details.

Adding Hand Engine Support to Your Scene and Linking Your Assets Hands

1. Import your character asset with articulated hands into your scene. We will use Kyle Robot from Unity Technologies for this example, but the process is identical for any asset.

2. In your Project Hierarchy:

1. Within your Assets folder, create a folder to contain the files that will be associated with your retargeting configuration files.

2. Navigate to the folder you created, right-click in the file list pane, then select Create → StretchSense → Hand Engine Configurator. We recommend you rename this prefab to a unique name of your choosing.

3. Drag and drop your newly created Hand Engine Configurator prefab into your scene. There are two key components in this prefab:

1. Hand Engine Client: Manages the connection to Hand Engine and receives hand animation data that can be used in your scene. You should only ever have 1x Hand Engine Client in your scene.

2. Retargeters: A prefab with two StretchSense RetargetHand script components attached to it, one for the left hand and one for the right hand. Additional RetargetHand scripts can be added here if you want to control more hands/objects with your StretchSense studio gloves.
   
   TIP: We recommend creating a prefab variant of the Configurator for each scene you want to add

3. In your scene Hierarchy, right-click on the Retargeters object in the Hand Engine Configurator and select Properties... to open a new Inspector pane with the Configurator options, then:

1. In your scene Hierarchy, expand your character asset until you can view the GameObjects corresponding to the left wrist joint. Each character is different but it is common for the name of this joint to include “wrist” or “hand”.

2. Click and drag the left wrist joint GameObject into the Wrist Root field for the Retarget Hand script in the Retargeters inspector pane that has the Source Hand set to LEFT. The text underneath the Wrist Root field will update with the name of the root parent of the GameObject added to Wrist Root so you can check the desired character is being referenced.

3. Click Create List next to the Target Joint Name list to create the list of target joints.

4. Find the GameObject corresponding to the right wrist joint, drag it into the Wrist Root field for the Retarget Hand script in the Retargeters inspector pane that has the Source Hand set to RIGHT, and click Create List next to the Target Joint Name list to create a list of target joints.
   
   TIP: The Target Joint Name List provides the context for selecting your bone names when configuring the Hand Asset Mapping (see below) so you don’t have to type out the names.
   

Using Simple Mode

Simple Mode uses 7 Key Reference Poses to retarget the performer’s hand motion onto the equivalent hand shape on your target character independently of the anatomical differences between the performer and character or the way the digital character is rigged. Retarget+ uses these key poses to determine the range of motion of each joint, and at the joint level blends between these boundaries using the control rig output of Hand Engine.
A description and illustration of each Key Reference Pose is provided in the table below.
 
TIP: All poses should be adjusted so they look natural on the character. Retarget+ works independently of any rotation order or default axes alignment that your character was created with. There is no need for rotations to align with any particular axes.
Table of Key Reference Poses for Simple Mode
  

Creating Your Reference Poses for Simple Mode

If you do not already have a hand pose library for your character, the following steps show a working example of creating 7x Key Reference Poses from scratch for use in Simple Mode.

If you do have a hand pose library for your character, you can follow along with these instructions and instead of manually posing the finger joints for each pose, you can apply the pose from your library instead (e.g. by setting the current animation of a static pose in the character model using an AnimationController).

For each pose, focus on making the pose look natural for the character’s hand. Retarget+ will take care of the mathematical transformations required to create precise blend spaces for animating the character hand.

The following instructions illustrate this process for one hand. Repeat these steps for any additional hands you are animating.
 

Before You Begin

Before getting started, it is helpful to pose your character's hands up with palm facing forward to make it easy to pose the hand, see the pose from multiple angles, and compare left and right-hand poses to each other to ensure symmetry.


 

Pose Creation Order

We recommend you start by creating the Paddle pose for your character as all of the poses can be derived from this pose. Once you have created a pose, click the Update button next to the relevant Reference Pose field in the inspector to store the pose information. You can easily reset the hand to that pose by clicking the Preview button next to that pose in the Reference Pose List.

We recommend the following order of creation:

1. Paddle pose

2. Paddle L pose

3. Paddle L Splayed pose

4. Reach pose

5. Paddle Thumb Curl pose

6. Paddle Thumb Curl Splayed pose

7. Fist pose

Paddle Pose

This pose captures the midpoint of the thumb movement when going from Paddle L to Paddle Thumb Curl

• Thumb tip straight and parallel to fingers, thumb base joints curled so thumb aligned with the side of the palm,

• Fingers straight and parallel to each other

Paddle L Pose

This pose captures the finger joint positions when they are straight with no splay.

• The thumb is straight, maximally opened out (i.e. maximum radial abduction), and approximately parallel to the plane of the palm.

• Fingers are straight and parallel to each other.

Tip: Start from Paddle and open the thumb out. Keep the fingers in the same position.




Paddle L Splayed Pose

This pose captures the rotation out of the plane of the palm of the base of the thumb when the thumb joints are maximally opened out.

• The thumb is straight, maximally opened out (i.e. maximum radial abduction), with the base of the thumb rotated forward to bring the thumb out of the plane of the palm.

• Tingers are straight and parallel to each other.

 
Tip: Start from Paddle L and rotate the thumb forward. Keep the fingers in the same position.

Reach Pose

This pose captures the maximum splay of the finger metacarpal joints, and the maximum opening out of the base of the thumb, and the thumb joint positions when the thumb is straight.

• The thumb is straight, maximally opened out (i.e. maximum radial abduction), and approximately parallel to the plane of the palm.

• Fingers are straight and maximally splayed apart.

Tip: Start from Paddle L and splay the fingers at the base. Keep the thumb in the same position.




Paddle Thumb Curl Pose

This pose captures the maximum curl of each of the thumb joints with the thumb against the surface of the palm.

• All thumb joints maximally curled, thumb on the surface of the palm.

• Fingers are straight and parallel to each other.

Tip: Start from Paddle and curl the thumb towards the pinky finger. Keep the fingers in the same position.




Paddle Thumb Curl Splayed Pose

This pose captures the maximum rotation out of the plane of the base of the thumb (i.e. maximum palmar abduction) when the thumb joints are also maximally curled.

• All thumb joints are maximally curled, plus the base of the thumb rotated out of the plane of the palm.

• Fingers are straight and parallel to each other.

Tip: Start from Paddle Thumb Curl and rotate the base of the thumb out from the palm. Keep the fingers in the same position.




Fist Pose

This pose captures the maximum curl of each finger joint, and the maximum rotation out of the plane of the base of the thumb (i.e. maximal palmar abduction) and is the midpoint of the thumb movement when going from Paddle L Splayed to Paddle Thumb Curl Splayed.

• Thumb base joints are curled so the thumb aligns with the side of the palm, the thumb base maximally rotated forward out of the plane of the palm, plus the maximum curl of the thumb tip.

• All finger joints are maximally curled.

TIP: We recommend posing the fingers in a fist using a two-pass process: align the fingers, and adjust for contact/clipping.

1. Align the fingers:

1. Start by applying the maximum bend at the base of each finger (typically between 90-100 degrees) and then make adjustments to ensure each finger is parallel (i.e. not clipping into nor splaying away from its neighbor), and that the finger pads on the ends of the fingers are all pointing in the same direction. You may find you need to make adjustments on multiple axes to achieve this. Focus on making the fingers look natural, there is no need to worry about aligning the finger position with any of the rotation axes.

2. Apply maximum curl to the next joint after the finger base joint, and again fine-tune the joint rotations in all axes if necessary to ensure the fingers are together and parallel.

3. Apply maximum curl to the last joint in each finger and again fine-tune the rotations on all axes if necessary to ensure the fingertips are curled naturally.

2. Adjust for contact/clipping:

1. If the fingertips are clipping into the palm of the hand, try reducing the maximum bend at the base of the finger in question. Adjust the bend of the other joints in the finger as necessary.

2. If it is too big of a gap between the fingertips and the palm, try increasing the maximum bend at the base of the finger to bring the fingertip in contact with the palm. Adjust the bend of the other joints in the finger as necessary.

Defining the Hand Asset Mapping for Simple Mode

Once each of the 7x Key Reference Poses have been created, it is time to create a Hand Asset Mapping to tell Retarget+ the names of the finger joints in your character.

Click the Update button next to the Hand Asset Mapping field in the HAND ASSET MAPPING section of the inspector pane. You will be prompted to specify a save location within your project for the scriptable object that will be created to store this information.

For each of the applicable joint fields in the HAND ASSET MAPPING section, click the “…” button to bring up a context menu with the names of the joints in your character’s hand to select from, which is populated from the Target Joint Names list we created earlier.
 
The Thumb CMC appears twice in the list of joints. This is by design, please select the base thumb joint for both fields.
 
Glossary:

• CMC: Carpometacarpal joint

• MCP: Metacarpal joint

• PIP: Proximate interphalangeal joint

• DIP: Distal interphalangeal joint

 
Your one-time setup of that character’s hand is now complete!


If Hand Engine is running and a glove is connected and calibrated, enter Play mode in Unity to see the live results of your retargeting configuration.