We spoke with several VR game designers and developers about the important (and difficulty) of creating believable and immersive hands in VR.
The Importance of Believable VR Hands
VR hands are a strange thing to ‘get to grips with’ so to speak. Not only do they allow you to interact with the virtual world in front of you but they allow you to become part of it. As you shape yourself to fit into that world, it does the same to fit around you. In real life, you have limited strength and ability but VR gives you that rare opportunity to be better than real life; it lets you become an action hero.
“It was pretty clear right off the bat that it was important,” Kerry Davis, a Valve programmer that worked on Half-Life: Alyx, said in an interview “Even if you didn’t have a full-body avatar yet, to at least have hands so that you could sort of connect with the world and actually participate. We all have an innate desire to control the world we inhabit. As a VR user, you don’t simply want to sit back and let the story happen—you want this organic control over the world. “People really wanted to have hands.”
Half-Life: Alyx’s design encapsulates this. There’s this visceral sense of control you have covering your mouth when you come across Jeff or even that satisfying click you get from loading up ammo. Tristan Reidford, a Valve artist who also worked on Alyx, learned this previously when working on the Aperture Robot Repair Demo that shipped as part of The Lab.
“We didn’t have hands in that… we simulated the controller… that satisfied people’s desires to have representation,” Reidford said.
The most fascinating thing about VR hands is the spectacle itself, the representation of you. When using a controller, you can disconnect and understand that certain buttons equal certain actions. The uncanny valley nature of movement in VR makes it just close enough to true immersion to become distracting when it’s not.
The uncanny valley in this case represents that physical and mental border between what your eyes see in the headset lens and what you feel around you. As VR gets closer to real life, it also gets further away. There’s something distinctly recognizable, yet alien-feeling about this imitation of real life. VR is at its best when it’s immersive and compelling but not trying to lie to you. It offers you a fantasy and, in the case of Half-Life: Alyx, that alien feeling comes from somewhere else: Xen
“Now you don’t need as much of an abstraction so you can almost get an exact representation of reality,” Davis said. “It turns out it’s almost harder to do in VR.”
The complicated nature of emulating some sense of real-life movement means you often have to trivialize or exaggerate that movement. “In VR, your interactions are so close that your brain wants to fall back to actions you’ve known since you were a toddler,” Davis said. “We still have to put this interface layer in there and say we’re defining what the constraints of this virtual world are.”
You are constrained in two senses when in VR. There are the constraints of your movement—such as how far you can move in your actual room and how much you can lunge forward before colliding head-first into your dresser—and the constraints of the tech itself. There is this wonderful creativity that springs from necessity when creating games atop necessary limitations.
Oftentimes, a world has to be made less organic and genuine to feel real, as paradoxical as that sounds. The swinging of a sword feels natural but you don’t have to undergo a year of training before you can use it in VR. It throws a little bit of ‘real life’ out the window to provide a more fun and, ironically, more immersive experience.
Limitless Limitations
Over at Streamline Media, a small group working on their first real VR title. “Due to resolution issues they (the team) often went back to using just larger gestures and bigger levers,” Stefan Baier, COO at Streamline Media, said. “Smaller hand gestures didn’t reliably scale and made it not work well with the dev work we were doing for PS4 where you don’t have that input.”
When working with it, PSVR often has to be at the bottom of the barrel, technologically speaking. This means that whilst hand gestures are held back in some sense, the choice to fit certain actions in are made purely for the player experience. When you can’t show off the tech or provide hyper-realistic details, you only provide what’s necessary. A streamlined control system allows for more natural movement, even if it’s constrained.
“We are always limited by the hardware…Especially with these fairly new technologies…tracking will massively improve”, Christof Lutteroth, a senior lecturer in the department of computer science at Bath University, said. “There’s nothing fundamental that will prevent us from simply tracking our hands. It’s just a computational problem that’s harder to solve.”
Ultimately, we are always held back by the tech but this doesn’t mean we should stop and accept it as it is.
“From what I’ve seen…It’s definitely a big step forward. However, it’s still very rudimentary,” Lutteroth said when talking about the Oculus Quest’s finger tracking. “When you come to hand tracking, there’s quite a lot of error involved with machine learning… That’s very often data-driven.” Due to the varied nature of human fingers (that’s before mentioning those with disabilities) hand tracking technology is dominated by research into what an average hand is. Unfortunately, with the way that research works, it often gets caught up in implicit biases.
One very highly publicized case of these forms of bias is Microsoft’s AI chatbot, Tay. It was designed to pick up on speech and emulate it to make Tay’s speech sound authentic. Within a day, it was virulently bigoted. This is an interesting microcosm of how these biases set in. If the people you test are racist, your chat bot will be. If the people you test have all ten fingers, your hand designs will be.
This is why Davis and Reidford were so outspoken in our interview about the power of playtesting. Reidford spoke of finger tracking and the unique hurdles in making your movement feel both organic, yet slick.
“There’s this sliding scale where, on one side, it’s fully finger tacking and then, on the other side, 100% animation,” Reidford said. ”We had to find the balance there… as soon you put the player under any kind of duress…They just want to slam a new magazine in.”
Surpassing Limitations
There’s creativity to just existing in VR. Davis mentioned that testers told him “‘Yeah, I’m playing this. I don’t feel powerful… I feel like my normal everyday self… that’s not why I go into games,’” Davis said. “‘I go into games to feel powerful and skilled.’” Even though it emulates your actions, VR is so captivating due to its ability to emulate the wonderful and powerful. That uncanny valley is less prevalent when the situation itself is one that you choose to put yourself into. When you’re aware of your world and place yourself in there, you can forget all about the gear you’re wearing.
A fantastic example of this is Half-Life: Alyx’s gravity gloves. “In Half-Life: Alyx it was about having the hands feel so natural you don’t really think about them anymore,” Reidford said.
You hold your hands up, grab an item and pull it towards you, and hope it lands. As long as you’re close to where you should be, it will always land. The same design philosophy is at work with the doors in Alyx. They don’t function like a normal door. You put your hand up to the doorknob and your character just turns it automatically.
“The player can still turn the handle themselves if they want to but they don’t have to, all they have to do is reach out, make a fist, and the door is open,” Reidford said. “Over time, it looks so correct and it’s what they expect to happen that the player actually believes that they’ve done it. They think that they reached out and turned the handle themselves when really they didn’t. The game did it for them”
The genius of Half-Life: Alyx is that this notion of feeling like “the action hero” is deployed so effectively without making you overpowered. You can simultaneously get crushed by creatures, cower away from Jeff, and giggle at the falling physics of a head crab—yet you still get out there, load your weapon, and take down the bad guys. This is a testament to VR as a whole. There’s a spectacle to it that can only be accessed via your hands. From its early days and crude movements to the beginning signs of significant hand tracking, there’s this bustling sense of creativity that keeps pushing the industry forward.
When you stare into the dark lenses of a VR headset and find yourself staring back, you look around your environment, look down at your hands, then squeeze closed your fists and you become something grander than the every day. You become your own version of an action hero.
One of the things that inspires me to do the Voices of VR podcast is being able to talk with hundreds of people listening for the underlying patterns of experiential design. VR & AR are reflections of the human experience, and so any experiential design framework for immersive technologies should also describe aspects of our conscious experiences. I’ve found that Eastern Philosophies, Chinese Philosophies, Natural Philosophy, and Hermetic Philosophies have useful ways of describing qualitative aspects of the human experience through concrete metaphors that can be adapted into holistic experiential design frameworks.
In the early stages of design, VR allows architects to understand the layout and feel of space within their design, leading to streamlined design development and more efficient internal communication from the get-go. But beyond internal use at their firm, VR enables an architect’s client to immediately understand how the project will look and feel, allowing the client to better interpret the space, catch mistakes, and articulate their design preferences prior to a project breaking ground. If utilized during a pitch, VR can help generate new business for a firm, and later on in the engagement can also save an architect and construction team countless hours of work and thousands (if not millions) of dollars preventing change orders and added construction fees.
