Unreal Engine (UE) Plugin
Software Setup and Human Model Creation
Enable "SDK" in the software.
After successful calibration in XINGYING software, create a human model in Live mode or load a set of data with MarkerSet human model in Edit and play it in the software.
Plugin Installation and UE Settings
For the UE_4.26 engine, unzip the "NokovLiveLink1.XXX" plugin version for UE_4.26 and copy the entire plugin folder. Paste it into the UE_4.26 engine's path, which is typically "Epic Games\UE_4.26\Engine\Plugins" (the actual path depends on the software installation on your computer) . If using the UE_4.27 engine, please download the "NokovLiveLink1.XXX" plugin version for UE_4.27. The operation steps are the same as for UE_4.26. For the specific plugin version, please consult a technical engineer.
If using the UE_5.0 engine, please download the "NokovLiveLink1.XXX" plugin version for UE_5.0. For UE_5.1 engine users, download the "NokovLiveLink1.XXX" for UE_5.1. Operation steps are the same as for UE_4.26 and UE_4.27. Please consult a technical engineer to obtain the specific plugin version.
After copying the plugin folder for version 4.26 to the corresponding location, open the UE_4.26 engine and create a project. Go to "Edit—Plugins", check "Animation—Live Link" and "Motion Capture—NokovLiveLink", and restart the software to activate the plugin. The steps are the same when using UE_4.27 or UE_5.0 or 5.1 engines;
In UE4.26, select "Window—Live Link," and click on "Source—Nokov Live Link," set the ServerIP to a fixed 10.1.1.198, ensure the Up Axis is consistent with the setup in XINGYING, and click OK. Play XINGYING to receive motion capture data, and a green indicator light will illuminate;
Create an Animation Blueprint by locating a skeleton with a "Skeletal Mesh" in the menu bar , right-click on this Skeletal Mesh—Create—Animation Blueprint , and double-click to open the created Animation Blueprint. Right-click in the Animation Blueprint and search for "Live Link Pose," then double-click to open it ;
In the "Live Link Pose," select the name of the target MarkerSet in "Live Link Subject Name" to drive the motion of the model in UE4 with that Markerset, and connect "Live Link Pose" to "Output Pose" by dragging . On the right side, select Retarget—Retarget Asset—NokovLiveLinkRetarget Asset;
Ensure that the human MarkerSet in XINGYING software stands in a T-pose position, click the Compile button in UE4, and when the human MarkerSet in XINGYING starts to move, the model in UE4.26 will be driven and move synchronously;
If using UE_4.27, first copy the plugin folder for version 4.27 to the appropriate location, then in 4.27 select Window—LiveLink. Once successfully connected to Markerset, in "Live Link Pose," select the name of the target MarkerSet in "Live Link Subject Name" to drive the movement of the model in UE4.27, and connect "Live Link Pose" to "Output Pose" by dragging. On the right side, select Retarget—Retarget Asset—NokovLiveLinkRetarget Asset.
Make sure that the human MarkerSet in XINGYING software stands in a T-pose. Click the Compile button in UE4, and as the human MarkerSet in XINGYING starts to move, the model in UE4.27 will be driven and perform synchronous movements;
If using UE_5.0, first copy the plugin folder for UE_5.0 to the appropriate location, then in UE_5.0 select Window — Virtual Production — LiveLink. Once successfully connected to the Markerset, in "Live Link Pose," select the name of the target MarkerSet in "Live Link Subject Name" to drive the movement of the model in UE 5.0, and connect "Live Link Pose" to "Output Pose" by dragging. On the right side, select Retarget — Retarget Asset — NokovLiveLinkRetarget Asset;
In XINGYING software, ensure the human MarkerSet stands in a T-pose, click the Compile button in UE 5.0, and as the human MarkerSet in XINGYING starts to move, the model in UE 5.0 will be driven and perform synchronous movements.
Introducing Human Data with Skeletal Redirection into UE
Human data from XingYing can be directly imported into UE using plugins to drive the human model. However, when there is a significant difference between the model and the actual person's body, direct driving of the model can result in issues like foot sliding. In this case, binding and redirection through MotionBuilder can be used before driving the human model in UE with MotionBuilder's UE plugin. Alternatively, in XingYing, you can import the human model you want to drive, then enable the redirection feature, and after the human data is converted, the output data can drive the human model directly in UE without MotionBuilder. To use XingYing's redirection feature to drive a model, refer to the following steps:
First, you need to create a human model in Live in XINGYING software or load human data in Edit modes, use the model asset import feature to import an FBX or HTR model file, and perform skeletal redirection. For detailed operation steps, please refer to the section "Fifteen, Data Redirection" above;
After re-directing the human skeleton in XINGYING, in the settings turn on "Enable Redirected Data", open UE, import the model file with skeletal redirection done in XINGYING into UE, right-click in an empty space, select Blueprint Class , in the pop-up window's textbox input "Nokov", select “NokovLiveLinkRetargetAsset”, and click Select . Name the Blueprint Class, if not renamed, the class will default to the name “NewBlueprint”. At this point, you can see that we have successfully created a Blueprint Class with a cube icon in the content side menu.
Double-click on the Blueprint Class you just created, and you will see two options for "Self Adaptation" and "Use Translation" at the bottom of the popup window. The translation option is checked by default, and self-adaption is unchecked by default. If you are not using skeletal redirection to drive the UE model, check the self-adaptation option and uncheck the translation option, click Save, and then Compile. If you are using skeletal redirection, ensure the translation option is checked and the self-adaption option is unchecked; click Save, then Compile. Driving the UE model with redirected human data and checking the self-adaption option may cause abnormalities in the model's skeleton in UE.
To drive the UE model using the skeletal redirection feature and after unchecking the self-adaptation option in the Blueprint Class, you can refer to steps 5-8 above to drive the model. The only difference is that you need to select the name of the Blueprint Class with the cube icon (that has the translation option checked) from the Retarget—Retarget Asset drop-down menu on the right side of the window. After selecting it, click Compile. When XINGYING software plays and the MarkerSet starts to move, the model in UE will be synchronously driven.
If you are not using skeletal redirection to drive the UE model, in the Blueprint Class check the self-adaptation option, uncheck the translation option, and select the name of the Blueprint Class with the cube icon (that has the self-adaptation option checked) from the Retarget—Retarget Asset drop-down menu. After selecting, click Compile. When the XINGYING software plays and the MarkerSet starts moving, the model in UE will be synchronously driven.
When driving the UE model with redirected human data, if you haven't checked or forgot to enable the "Use Redirected Data" feature in XINGYING, you will need to turn on "Use Redirected Data" in XINGYING. After enabling it, disconnect the link between UE and XINGYING and re-establish the LiveLink connection. Otherwise, the inconsistency between the human data and the description information obtained by the SDK may lead to abnormalities in the model's skeleton in UE.
Blueprint Class
Double-click on the created Blueprint Class to enter it;
The Nokov skeleton Mapping table shows XINGYING's human skeleton names on the left and UE model skeleton names on the right. By selecting the checkbox for Enable skeletonMapping in the Nokov skeleton Mapping option, you can edit the list below, otherwise, it's not editable. There is a checkbox on the left side of each joint skeleton name; if checked, this joint skeleton is involved in data driving; if not checked, the skeleton will not participate in data driving.
Click the Select button to the right of an individual joint (row), and enter the skeleton name in the search box to fuzzily match the target joint. For example, if you select the "Spine1" skeleton and click the Select button, you can enter "Spine" in the search box to fuzzily match the target joint ;
Click "Select a skeleton for mapping" at the bottom of the window, which will display all the skeleton resources Select a skeleton resource to apply that skeleton and check if the skeleton names in it correspond correctly with the skeleton names of XINGYING's human model. If there's a mismatch, you can use the Select button to match the target joints via fuzzy search;
If you are using motion capture data to drive the UE model and the model's skeleton display abnormalities or incorrect positions, this may be due to some skeleton names being inconsistent with those in XINGYING software, leading to abnormal skeletons when driven in UE. In this case, you need to check the mapping of the skeleton names. Double-click on the created Blueprint Class, check the skeleton joints on both sides in the Nokov skeleton Mapping list to ensure they correspond correctly. If there's a mismatch, after making the necessary changes, save and click Compile. Then, select the name of the Blueprint Class we just modified in the Retarget—Retarget Asset drop-down menu and click Compile to successfully drive the model's movement;
To see the skeleton names and hierarchy of the model file imported into UE, double-click on the model, click on the skeleton icon in the model window, and you will see a tree diagram on the right side showing the skeleton names of the model ;
When selecting a part of the model's skeleton from the skeleton name tree diagram, the middle of the model will display the specific location of that skeleton on the model's body. This method is used to verify that the UE skeleton names correspond correctly with the XINGYING human model's skeleton names. For example, select the UE model's forearm skeleton, and check in the Nokov skeleton Mapping list if the UE forearm skeleton name corresponds correctly with the XINGYING human body's forearm skeleton. If you find that the UE forearm skeleton is incorrectly mapped to the XINGYING body's upper arm skeleton, copy the name of the forearm skeleton from the UE model and correspond it to the forearm skeleton of the XINGYING body in skeleton Mapping. Do this for each skeleton to ensure all names match correctly. After completing the checks, click Save. You can view the XINGYING human model's skeleton positions and their names by clicking the "Joints" list in the XINGYING assets panel;
If the skeleton are correctly mapped between XINGYING and the UE model, but the model still shows abnormalities in UE, please contact our technical engineers for assistance.
LiveLink Rigid Body Application
To use UE rigid body application, first place an object with reflective markers in the motion capture area, and in Live mode, use the one-click create rigid body function to create a Markerset. After successfully creating the rigid body, open the Soft Data Broadcast panel and enable "Enable SDK";
Open the UE project file. If creating the UE project for the first time, please contact our technical engineers to obtain the corresponding plugin version and refer to the above "1) UE Plugin" to correctly configure our plugin;
Click Window — Virtual Production — LiveLink in UE. In the LiveLink window, click to add a source, select "Nokov LiveLink". Keep the IP address consistent with the XINGYING software, defaulting to 10.1.1.198, and ensure the axis directions match those in XINGYING software, which defaults to Y-up;
After completing the settings, click OK, play the XINGYING software, and you will see a green light indicating that it is active, displaying the names of all connected Markersets. A character name of "Transform" represents that the MarkerSet is a rigid body, while a name of "Animation" indicates that the MarkerSet is a human;
Next, quickly create a static mesh in UE and add it to our project, select a geometry shape,
After creation, you can see the instance we just created in the "Details" property bar on the right, and the geometry sphere has been added to the view . Click the add button, enter "LiveLink", and select the "Live Link Controller" component;
After adding the LiveLinkComponentController component, select the component. In the dropdown menu on the right side of "Subject Representation," select the name of the rigid body that we connected earlier in the LiveLink tab, click on "Tracker0" , to use the rigid body named "Tracker0" to drive the geometry sphere we added in UE;
Select the root component of the default scene and change the component's mobility to movable ;
Next, just play the XINGYING software, and as the "Tracker0" rigid body moves, the sphere in UE will move in synchronization with the rigid body.
Solutions for UE Engine Refresh Issues
When using the XINGYING plugin to drive a model in UE, if XINGYING loses the human body and re-recognition does not display the model in the UE scene, this phenomenon is caused by a refresh issue in the UE engine. To solve this refresh problem, after linking the Live Link Pose to the Output Pose to drive the model in the Animation Blueprint, you also need to add nodes in the event graph of the Animation Blueprint and connect the added nodes. Doing this can solve the problem of the UE engine not refreshing.
Last updated
