Unreal Engine Plugin

（1）Software setup and body creation

Open Settings in the software - select the "Network" TAB - select "Use SDK";

After successful calibration in XINGYING software, create a mannequin in real-time mode or load a section of data with a Markerset human body in post-processing.

（2）Plug-in installation and UE4, UE5 Settings

To use the UE_4.26 engine, unzip the "NokovLiveLink1.XXX" plugin version of the UE_4.26 engine and copy and paste the entire plugin folder into the path of the UE_4.26 engine.The specific path is "Epic games\UE_4.26\Engine\Plugins", depending on the software in your computer (Figure 220).If the UE_4.27 engine is used, you can download the NokovLiveLink1.XXX plug-in of the UE_4.27 engine. The operation procedure is the same as that of the UE_4.26 plug-in (as shown in Figure 221). For details about the plug-in version, contact technical engineers.

Figure 220

Figure 221

If you use the UE_5.0 engine, please download the "NokovLiveLink1.XXX" plug-in version of the UE_5.0 engine, and use the UE_5.1 engine.Please download the "NokovLiveLink1.XXX" of the UE_5.1 engine in the same way as that of UE_4.26 and UE4.27 (as shown in Figure 222). For details about the plug-in version, contact technical engineers.

Figure 222

Open the UE_4.26 engine after copying the 4.26 plug-ins folder to the appropriate location, and create the project. Select "Edit - Plug-in", check "Animation - LiveLink" and "MotionCapture - NokovLiveLink" (as shown in Figure 223 and 224), and restart the software for the plug-in to take effect.If the UE_4.27, UE_5.0, or 5.1 engine is used, the steps are the same as those for UE_4.26.

Figure 223

Figure 224

Select "Window - LiveLink" in UE4.26, click "Source - Nokov LiveLink", SDKVersion select V2, ServerIP fixed at 10.1.1.198,,Select the same Euler Angle rotation order as in the software (the default Euler Angle rotation is "ZYX", if there is no adjustment in the software, it will remain unchanged here), and the up axis (the up axis set in UE is the same as XINGYING), and click OK. Play XINGYING, receive mobile capture data, and the green indicator lights up (Figure 225);

Figure 225

To create an animation blueprint, find the skeleton with "skeleton grid body" in the menu bar (see Figure 226), right-click the skeleton grid body-Create - animation blueprint (see Figure 227), double-click the created animation blueprint to open it.Right-click in the animation blueprint to search for "live link pose" and double-click to open it (Figure 228);

Figure 226

Figure 227

Figure 228

In the "Live Link Subject Name" of "Live Link Pose", select the name of the target Markerset to drive the model movement in UE4, and drag the "Live link pose" to the "output pose" (see Figure 229).And select Retarget - Retarget Asset - NokovLiveLinkRetarget Asset on the right side (see Figure 230);

Figure 229

Figure 230

Make sure that the human Markerset in XINGYING software is standing in T-pose, click the compile button in UE4 to make the human Markerset in XINGYING move, and the model in UE4.26 will be driven and move simultaneously;

If UE_4.27 engine is used, please copy the plugin folder in 4.27 to the corresponding location and select LiveLink (as shown in Figure 231) from 4.27. After successfully connecting to Markerset, click LiveLink Subject Name under LiveLink Pose.Select the name of the target Markerset to drive the model motion in UE4.27 and connect the Live Link Pose to the Output Pose drag.On the right, select Retarget - Retarget Asset - NokovLiveLinkRetarget Asset;

Figure 231

Make sure that the human Markerset in XINGYING software is standing in T-pose, click the compile button in UE4 to make the human Markerset in XINGYING move, and the model in UE4.27 will be driven and move simultaneously;

If the UE_5.0 engine is used, please copy the plug-in folder of UE_5.0 to the corresponding location and select the window - Virtual film - LiveLink(as shown in Figure 232) in 5.0. After successfully connecting to Markerset, click "LiveLink Subject Name" under "LiveLink Pose".Select the name of the target Markerset that drives the model motion in UE 5.0 and connects the Live Link Pose with the Output Pose drag.On the right, select Retarget - Retarget Asset - NokovLiveLinkRetarget Asset;

Figure 232

Make sure that the human Markerset in XINGYING software is standing in T-pose, click the compile button in UE 5.0, and make the human Markerset in XINGYING move, and the model in UE 5.0 will be driven and move simultaneously.

（3）Bone redirects human data into UE

XingYing's human data can be directly connected to UE-driven mannequins through plug-ins. However, when the body difference between the model and the model is large, the model can be directly driven or slippage occurs. In this case, the UE plug-in of MotionBuilder can be used to drive the UE human model after binding and redirection. You can also import the mannequin that needs to be driven in XingYing, and then turn on the redirection function. After the human data is converted, the output data can skip MB and directly drive the mannequin in UE. To drive the model through XingYing's redirection function, you need to refer to the following steps:

First, you need to create a human model in real time in XINGYING software or load human data in post-processing mode, and use the model asset import function to import FBX or HTR model files for bone redirection. For detailed operations, please refer to "Model Asset Import" above.

After completing the redirection of human bones in XINGYING, open the software Settings, select the "System" TAB, check "Enable redirection data" (as shown in Figure 233), open UE, create a new project, import the model file of bone redirection in XINGYING into UE, and click the right mouse button in the blank area.Choose blueprint kind (as shown in figure 234), in the pop-up window in the input box input "Nokov", select "NokovLiveLinkRetargetAsset", click options (as shown in figure 235), to the blueprint, after class if not rename, blueprint for the default name for "NewBlueprint",At this time, you can see that in the content slider menu, we successfully created a blueprint class of cube ICONS;

Figure 233

Figure 234

Figure 235

Double-click the blueprint class just created, you can see that there are two options at the bottom of the pop-up window: Self Adaption and Use Translation. The translation option is checked by default, while the adaptive option is not checked by default (as shown in Figure 236). If the UE model is not driven by bone redirection function,Please check the adaptive function, uncheck the translation function, click Save, click compile.If the bone redirection function is used, please check whether the translation option is checked. If the adaptive option is checked, please uncheck it, click Save, click Compile, and use the human data of bone redirection to drive the UE model. If the adaptive option is checked, the skeleton of the UE model may be abnormal.

Figure 236

After the bone redirection function is used to drive the UE model and the adaptive function is unchecked in the Blueprint class, the model can be driven according to the steps in "Plug-in Installation and Setting of UE4 and UE5" above. The only difference is thatYou need to select the blueprint class name of the cube icon with translation option checked in the drop-down box of Retarget-Retarget Asset on the right side of the window (as shown in Figure 237). After selection, click Compile, play the XINGYING software, and Markerset will start to move, and the model in UE will be driven synchronically.

Do not use the bone redirection function to drive UE model, please check the adaptive function in the blueprint class, uncheck the translation function, and select the blueprint class name of the cube icon with the adaptive option checked in the drop-down box of Retarget - Retarget Asset, and click Compile, XINGYING software to play.When the Markerset starts to move, the models in UE are driven simultaneously.

Figure 237

The UE model is driven by bone redirected human data. If the function "Start redirected data" is not checked, it needs to be checked. At the same time, disconnect UE from XINGYING software and start LiveLink again, otherwise, the human data will be inconsistent with the description information obtained by SDK.This results in bone abnormalities in UE model.

Double-click the created blueprint class (Figure 238);

Figure 238

In the Nokov Bone Mapping table, the left side is the bone name of XINGYING's human body, and the right side is the bone name of UE model. In the Nokov BoneMapping option, check the Enable BoneMapping check box to edit the list below. If it is not selected, the list cannot be edited. There is a check box on the left of the name of each joint skeleton.If it is not checked, the bone will not participate in data drive (Figure 239);

Figure 239

Click the Select button on the right of a single joint (row), and enter the bone name in the search box to fuzzy match the target joint. For example, Select the bone "Spine1", click the select button, and enter "Spine" in the search box to fuzzy query and match the target joint (as shown in Figure 240).

Figure 240

Click "Select a skeleton for mapping" at the bottom of the window, and all skeleton resources will be displayed (as shown in Figure 241). Select skeleton resources to apply this set of skeleton. Check whether the skeleton names in the selected skeleton resources correspond to those of XINGYING's human body.The target joint can be fuzzy matched in the Select button.

Figure 241

If the UE model is driven by dynamic capture data, the bone display of the model is abnormal or the position is wrong, which may be due to the incorrect correspondence between some models and the bone names of XINGYING software, resulting in abnormal bones after driving the model in UE. At this time, we need to check the name mapping of the bones and double-click the blueprint class we created.Check whether the skeletal joints on both sides correspond correctly in the Nokov Bone Mapping list. If they correspond incorrectly, save the modification and click Compile. Select the blueprint class name we just modified from the drop-down box of retarget-Retarget Asset, and click Compile to successfully drive the model movement.

To view the bone name of the model imported into UE, you can double-click the model and click the skeleton icon in the model window, and you can see that the bone name of the model is displayed in the tree diagram on the right side of the window (see Figure 242).

Figure 242

Select a bone of the model in the bone name tree, and the middle model will show the specific position of the bone on the model. In this way, check whether the name of the UE bone corresponds correctly to the name of the XINGYING human bone. For example, if the UE model forearm Bone is selected, check whether the UE forearm bone name corresponds correctly to the forearm bone of XINGYING in the Nokov Bone Mapping list. If the UE forearm bone corresponds to the upper arm bone of XINGYING, the forearm bone name of UE model needs to be copied.In Bone Mapping, the name of the copied forearm bone corresponds to that of XINGYING's human forearm bone, and so on. Check whether the name of each bone segment corresponds correctly one by one. After the check is completed, click Save. The location of individual bones of XINGYING and their names can be viewed by clicking "Segments" property bar on the right side of the capture software.

If the bones of XINGYING are correctly matched with the bones of UE model, the bones of the drive model still show abnormal in UE, if such a situation occurs, please contact technical engineers for consultation.

（4）LiveLink uses a rigid body

To use UE rigid body application, first put a capture with a reflective marker in the motion capture field, and create a Markerset using the one-click Create rigid body function in real-time mode. For details, please refer to the software manual. After the rigid body is successfully created, open the software Settings, and select "Use SDK" on the "Network" TAB.

Open UE project file. If you create UE project for the first time, please contact our technical engineers to obtain the corresponding plug-in version, and refer to "Plug-in installation and UE4,UE5 Settings" above to configure our plug-in correctly;

Click the window - Virtual film-livelink in UE (as shown in Figure 243), click Add source in the TAB, select "Nokov LiveLink", change "Nokov SDK Version" to V2, and keep the IP address consistent with XINGYING software, the default is 10.1.1.198.The axis is consistent with the axis of XINGYING software, and the default Y-axis of the software is upward (Figure 244).Choose the same Euler Angle rotation order as in the software (the default Euler Angle rotation of XINGYING human Markerset is "ZYX", if there is no adjustment in the software, it will remain unchanged here);

Figure 243

Figure 244

After all Settings are completed, click OK to play the XIINGYING software. At this time, you can see that the green indicator on the TAB will light up and display the names of all connected Markersets. The role name "Transform" means that the Markerset is a rigid body.It means that the Markerset is a human body (Figure 245).

Figure 245

1 Next, quickly create a static mesh body in UE to add to our project, select a geometry (Figure 246),After the creation, the instance we just created can be seen in the "Details" property bar on the right. The geometric sphere has been added to the view (as shown in Figure 247). Click the Add button, enter "LiveLink", and select the component "LiveLink Controller" (as shown in Figure 248).

Figure 246

Figure 247

Figure 248

After adding LiveLinkComponentController component, select the component, in the bottom of the "Subject Representation" to the right of the drop-down box to select our above LiveLink TAB in the rigid body connected by name,Click "Tracker0" (Figure 249) and use the rigid body named "Tracker0" to drive the geometric sphere we added in UE;

Figure 249

Select the root component of the default scenario and change the component's mobility to moveable (Figure 250);

Figure 250

Next, you only need to play XINGYING software and make the rigid body "Tracker0" move, and the sphere in UE will move synchronously with the rigid body.

（5）Solution to the UE engine refresh problem

When the XINGYING plug-in is used to drive the model in UE, the XINGYING human body is lost, and after the human body is re-identified, the model cannot be displayed in the UE scene, which is caused by the refresh problem of the UE engine. To solve this refresh problem, first connect the real-time link pose and output pose to the driver model in the animation blueprint.You also need to add nodes to the event diagram of the animation blueprint, and connect the added nodes (Figure 251), so that the UE engine does not refresh.

Figure 251