Fidelity & SuperSplay Gloves - Unreal Engine Streaming

Requirements

• StretchSense Hand Engine: Login to your account page at http://www.stretchsense.com/my-account  or email support@stretchsense.com for your license details.
• Unreal Engine
• Unreal Engine - Hand Engine Plugin 

• Download the most up-to-date Plugins from the Downloads Section of your StretchSense Account at stretchsense.com/my-account/

1. Set Up the Remap Tool in Hand Engine

Remapping is the function of assigning the bones of the hand asset within Hand Engine to drive the correct bones on the asset you are streaming hand data onto in another software. 

IMPORTANT: This task will need to be completed prior to streaming hands to Unreal Engine using the Fidelity glove.

Set up the T-Pose of your rigs FBX

In order to remap the Hand Engine output to your custom character, you must first create an FBX that has just your target character in it and that character is in a full T-Pose that is saved at the first frame in the FBX.

The character must be in a full T-pose, with each hand in an L-Pose. The fingers of the character should align with a major axis of the scene/FBX.

Please note this FBX is used only for extracting bone hierarchy and rotation order data from your character. Once the remapping process is complete, you will be able to stream data from Hand Engine to any FBX/scene containing that character.
The key features for each hand pose are:

1. The palm is facing down and is parallel to the ground plane
2. The fingers are pointing straight out and are parallel both to each other and to the ground plane, and are aligned with a major axis of the scene/FBX e.g. the X-axis in the screenshots below.
3. The thumb should be pointing straight ahead, perpendicular to the fingers, and parallel to the ground plane. It should be aligned with one of the other major axes of the scene/FBX, such as the Z-axis in the provided screenshots. Additionally, the thumbnail should be facing towards the body, which means it should be pointing in the opposite direction to which the fingers are pointing.

NOTE: The Hand Engine hand asset is set up using the metric system. Please ensure that your target FBX is set up with the same measurement system (working units in centimeters) to the software you are streaming.

Example of correct bind L-Pose

Common Mistakes

The thumb is in a relaxed position. This will cause the thumb to be tucked in too close to the hand

Fingers are curled/in a relaxed position. This will cause the fingers to be over-rotated then creating a fist.

 

Fingers and thumb are correctly set, but the hand/arm is rotated relative to the major axis of the global scene. This will cause off-axis rotations of the fingers/thumb when curling them.

 

Creating a remap profile in Hand Engine

1. Open the remapping tool by going to Tools -> Remapping
   
2. Select the character FBX you have set up as per the above instructions:
   
3. Once the file has been selected, please select the root of the left hand from the left drop-down menu(1), or you can use a previously saved Remap profile(2). 
   
4. Hand Engine will detect the remainder of the bones on the hand hierarchy and list them below. 
   
   
   This is how the Hand Engine asset is mapped. Please use this as a guide for assigning the bones of your asset to the bones of the hand in Hand Engine:
   
   
    
5. The right-hand column contains the hand bones of the asset within Hand Engine. Please select the associated bone to match your character's hand bones from this list. 
   
6. Once this has been completed, you can adjust the save name of this mapping at the top right-hand side:
   
7. To save this remapping, click on submit at the bottom of this window:
   

If you need to make any adjustments to this remapping, you can make the changes and click update at the bottom. It will overwrite the previous saved remap:

 

Assigning remap profile to a hand

1. Please connect gloves and perform a calibration
2. Click on Remapping Target in the hand setup box, and select your recently added remap. Make sure to add it for each hand.
   
3. Once this has been done for each hand, the hands are now ready to stream outside of Hand Engine to your software of choice. 

NOTE: If you have created or updated a remapping profile, after selecting the remapping target for each hand, please turn the TCP stream OFF and then ON again to make sure the new remapped stream is active. 
 

2. Setting up your Unreal Project

1. Download and Install Unreal
   NOTE: for this example, we are using version UE5.1
    
2. Next, we need to add the plugin to the Unreal Engine file directory. To do this, first, navigate to the folder you have downloaded the Unreal plugin from StretchSense, and find the plugin directory corresponding to the version of UE4 you are running
3. Open the folder corresponding to your version of Unreal, then copy the folder labeled MocapProLiveLink to the directory:

 <Unreal Installation Directory>\UE_4.2X\Engine\Plugins\Animation

4. Next, we will create a new UE project. In Unreal Project Browser select Games > Blank, then click Create Project.

5. Once the project is loaded we need to enable the LiveLink plugins in your new project. From the Edit menu, select Plugins.
   

6. In the Plugins menu search “Live Link” and enable Live Link and Mocap Pro LiveLink, then restart your project when prompted.
   

7. Next, open the LiveLink Menu from Window > Virtual Production > Live Link
   
    
8. From the Source dropdown in the Live Link menu select Mocap Pro Glove, then set the IP Address and TCP Port settings as specified in Hand Engine. You will have to add one live link to each hand.
   

NOTE: The default value for Address is 127.0.0.1 (= localhost), i.e., you are running Hand Engine on the same PC that you are running Unreal. If you have bound Hand Engine to a different IP address (e.g. if you are running Hand Engine and Unreal on different computers on the same LAN), use this IP address instead for Host.
 

NOTE: This step must be repeated if you have toggled Streaming to Off and then back to On in Hand Engine.
 

9. The Source Status will appear as Receiving and the Subject Name will show a hand (left or right) along with the Performer Name from Hand Engine.
    

10. From the Content Browser select Import and load your custom rig that has been set in a T-Pose as instructed in the beginning of this guide.
    

11. In the Content Browser, right-click the newly imported Skeletal Mesh and select Create > Anim Blueprint.
     

12. Open the Anim Blueprint that was created, right-click on the space, and select Live Link Pose.
     

Connect a LiveLink Pose to the Output Pose. Select the source hand from the Live Link Subject Name dropdown, then compile the object. 
 

13. You should now be able to see finger data streamed into your customer character.