Human upper-body inverse kinematics for increased embodiment in consumer-grade virtual reality

As a gaming and tech enthusiast, I had to get my hands on a VR headset. Doing a master thesis in VR was a great way to justify that purchase back in 2017.

In this thesis/paper, I experimented with upper-body animation, i.e., animating arms (and optionally shoulders) using the hand and head pose as input. This proved to be much harder that I expected. Off-the-shelf IK (inverse kinematics) solutions just didn’t feel natural. So, instead, I hacked together a solution that provides better results for this specific task - still not perfect, but better. If you know the distance between the user’s controllers in a T-Pose and the height of the headset when standing straight, you can create much better pose estimations. But of course, it is still very easy for the arms to move incorrectly. For example, when you simply rotate your elbow up and down without moving the hands or the head. That’s impossible to solve without additional inputs.

So, in order to measure how good these arms felt for the users - and if they can even be used for interaction in VR, I created a user study in VR. The users would play a round of archery, a sport in which you always see the bow holding arm, but the arm itself is not used for interaction. Afterwards, they would play a game where colored balls were flying towards you and you have to hit them with the body part of matching color - left arm, left hand, right arm, right hand. Surprisingly, the users preferred our solution over the “correctly” tracked motion capture arms. To be fair, the motion capture setup added notable lag over our solution.

While this is only research code and definitely not production ready, it turned out to be useful for many (hobby) VR developers with >3k downloads in the Unity Asset store. (Kind of embarrassing that I still didn’t invest the time to fix some of the broken math that was just about good enough for the study, but not for production.)