Creating a Hand Tracking VR Experience
Updated: May 5
10 Steps with Downloadable Checklist
We spoke with numerous hand tracking VR developers from around the world to gain insight about the struggles and successes of creating a full hand tracked VR experience for the Meta Quest. Based on our conversations, we developed a checklist for how to design, develop, and successfully launch a hand tracking VR title from the ground up. From building your team to your building out hand gestures and mechanics, to getting a title published and bringing in sales––here’s everything you need to know to get started.
Featured below is Jordan McGraw, CEO of Nooner Bear Studio and creator of the viral Meta Quest hand-tracking title, Rogue Ascent, to gain further insight into what it takes to make a hand-tracking VR game successful.
1. Ask the Question “Why am I making a hand tracking game in the first place?”
If you are thinking about designing a VR experience that utilizes hand tracking it is essential to “know your why”. There is no one right answer to this question, but you better make sure you have your answer – and can defend it to yourself or anyone else who might not see your vision.
With hand tracking VR experiences, it is a “all or nothing” mentality. Hand Tracking is not just another feature that can be fully utilized by plugging into existing controller-based games. Hand tracking must be the foundational technology in which you build your VR experience.
Hand tracking is a design challenge for many reasons. For an ambitious developer, this challenge alone may be incentive enough. Maybe you have a burning design idea you want to investigate. Either way, hand tracking should not be seen as a trivial analog to controller-based input. There will be new challenges and little precedent to draw from. For these reasons it is important to have a strong grasp of why you are setting out to design a hand tracking VR experience.
Jordan’s Thoughts:
"To take an existing controller-based game and implement hand tracking does little service to the technology and to your player. Hand tracking must be treated as a foundational technology. After all, if hand tracking can only measure as tall as the controller it replaces, why bother at all? We have to bring something new to the table. This is an emerging technology– we’re all out here figuring it out together. There’s no shortage of trailblazing involved. You’ll need to get comfortable working in a design space without any handrails. Your job as the game’s designer will be to utilize all of your creative resources to their fullest capacity. Take full advantage of lessons learned in fields of psychology and physiology and build upon them."
2. Build the Right Team
To successfully design and create a hand tracking VR game, you’ll need to put together the right team. Ideally, your team members also have experience in designing VR games using hand tracking technology. If they do not, it is essential that are fully bought into your “Why” (see point #1 above), and share the same passion for technological trail-blazing as you do.
Your team will essentially define and plan your hand tracking project. For a typical indie team to do this, you’ll need to hire a project manager to lead the team. The rest of the team can be filled with some or all of the roles listed below. Note that you will not need a single individual for each role. With VR design studios, oftentimes a small team of 3 or 4 can wear all necessary hats to complete the project.
VR developers
UI designers
Game Artists / Illustrators / Graphic Designers
Business Analysts
VR Publisher (see point #7)
Web developers
Mobile developers
Quality Control Managers
Testers
DevOps engineer
Jordan’s Thoughts:
Find people who understand your vision. Always surround yourself with those who understand your needs as a developer. As a member of the team, you need to create a dialogue which facilitates creative thinking. Be open and communicative with your team about what is needed, and provide the wider context where their work will fit in. Don’t wait for gratitude to go on sale– give it to your team when it’s most valuable.
3. Make Hand Tracking a Value-Add, Not a Gimmick
With any restrictive or complex design, you have to align your perspective with that of the technology itself. What is it capable of achieving? What are its limitations? Are there gaps in the technology to be addressed? Be honest and reasonable in your estimation. Understanding the limitations of a technology is not a drawback. Rather, it creates a foundation of strength in your design. Aim for simplicity in all things.
Abide by the Golden Rule of Hand Tracking: The experience must feel better with hands than controllers….otherwise it is a useless gimmick.
Jordan’s Thoughts:
You’ll always hear me talk about the field of Human Factors. Human Factors is the field of inquiry that informs an engineer how far to raise the volume of factory alarms above the ambient decibel of the shop floor. It’s the field that gives designers the intuition to design ease-of-motion interfaces and the insight of when to use analog or digital instrument displays in an airplane cockpit. It’s everywhere. Human Factors is the reason your browser’s exit button occupies a corner where it can’t be overshot by your cursor. Every good design will have its roots in Human Factors. Find and water those roots.
4. Get Creative with the Different Orientations and Gestures for Your Hands
Don’t just take controller-directed gestures that have been in video games for years and make them hand-tracked. This is a whole new world that has yet to be explored with countless new gestures and movements that can really mean something within the game.
Consider the psyche of the real-life player…Their entire gaming universe has revolved around moving joysticks and pushing buttons. Now they can actually BE their characters. They can make their characters MOVE how THEY MOVE. Use this to your advantage to create a new level of immersion that no one has ever experienced before.
Jordan’s Thoughts:
There are always more factors to consider than you will initially bring to bear on a problem’s solution. Take time to think orthogonally. Let’s pretend you’re having difficulty with a unidimensional input system– are you leveraging the temporal dimension? Build a comprehensive input space. What designs are people already familiar with? What existing designs can be adapted to match your new method of interaction? You can always unfold narrow concepts into higher dimensions. Do not accept limitations at face value.
5. Don't Over-Complicate the Controls & Keep Everything Fluid
Your player will thank you for keeping things simple. What is the player’s ‘home-base’ for gesture controls? Can the player relax their hand without initiating an action? How do gesture actions overlap? What priority do you assign to gestures if they overlap? How can you utilize gesture overlap? Consider the musculature of the human hand and always aim to reduce the amount of movement a player is required to make.
Jordan’s Thoughts:
Physical fatigue is your foremost consideration. Is a gesture easier to reach from one position than it is from another? A hand gesture may be easy while sitting at your desk, but can you repeat that gesture for 30 minutes straight? Develop gestures such that their activation thresholds can be adjusted. Don’t exhaust your player by having them hold their arms out for extended periods. The present force of gravity is always taxing your player’s endurance.
6. Discuss Accessibility for Those with Disabilities
How and should you include it in your hand tracking VR experience?
Accessibility is the willingness to create opportunities so that people can overcome obstacles in daily situations. In the context of designing a hand tracking VR game, accessibility is the pursuit of ensuring access and inclusion for people with disabilities within the UX design of the experience itself. Promoting digital inclusion is not only the ethically right thing to do, but it also creates conditions where more users can use your product.
Jordan’s Thoughts:
Always keep the plates of accessibility spinning during development. You and your testing environment will necessarily represent an incomplete picture of the larger population. Think over the most simple ways to remain inclusive of players who may face additional challenges utilizing your designs. Seek out feedback from those who can provide insight about disabilities and how they overcome them. Make Human Factors your friend.
7. Choose the Right Publisher
Managing a company is a challenge no matter how you slice it. You should work to understand the role of a publisher and their motivations as an entity.
Publishers will often focus on a specific market. Do they specialize in mobile games? PC titles? Virtual Reality? How about specific genres? Are they known for story-driven games? What does their history of experience look like? Does it make sense to bring them on for your title? What connections do they have to larger distribution platforms?
A publishing team is there to provide you with the resources you need as a developer to bring your vision to life.
Jordan’s Thoughts:
A strong publishing team will be deeply invested in your project and your growth as a professional. There should be prompt, clear, and productive communication between you and your publishing team. This is your project. Allow them to advise you– it’s their job. Avail yourself of the value they provide. Your communication should reflect a mutual respect and understanding of each other’s role. If a publisher did not meet any of these criteria you would not see me working with them.
8. Find your Platform(s)
The big thing here is to make sure you build for platforms that can support the hand tracking feature. There are others, such as the Magic Leap and Vive One, but the best bet is to design your hand tracking VR game or experience for the Meta Quest Platform, which is the most utilized hand tracking platform in the world.
Jordan’s Thoughts:
Each platform allocates their strengths differently. Have respect for yourself and your product when selecting platforms. Is exclusivity something you welcome or aim to avoid? Does the platform have a promising future? Are they willing to provide development units for your team? Will they help fund development? Research their market share. How long will the platform maintain their hold on the industry?
9. Use Marketing to Effectively Show This is a Hand Tracking Title
Since this is still a new technology, it will take some time for people to immediately recognize a hand tracking game when they see it. It is you and your publishers job (see point #7) to effectively and creatively communicate your unique value proposition and competitive differentiators.
You need to demonstrate that the actions being shown are not a result of traditional VR input. This may require adding special features which are only used for demo purposes. Get creative with your pitches and show why you believe hand tracking is the future of gaming.
Jordan’s Thoughts:
Fair or not, there’s a lot of weight on your shoulders to get this right. Be willing to receive criticism from those who shy away from what they consider unproven technology. Be the one to prove them otherwise. Many people who discouraged our efforts were those who had never actually used the technology before. You’d be surprised how hard people will knock things before they try it.
10. Plan for Post-Launch
Your Game is Live but Don't Forget to Keep the Community Alive.
Building a community around your game can be tricky. But keeping that community alive and growing AFTER you launch your hand tracking game can be even more challenging. A few ideas to keep cultivating your community are below:
Create a Discord for your game and entice users to join by giving them:
Free stuff
Exclusive stuff
A feeling that they are part of something bigger than themselves
Be transparent with your development of the game
Livestream development
Regularly post bugs/fixes/updates/ideas
Create & Aggressively build a mailing list, then creatively keep them in the loop
Use your social channels to keep new interest and to invite users to join the discord server
Jordan’s Thoughts:
Play is an primordial behavior originating long before the advent of organized games. Social play in particular is one of life’s true rewards. Even animals seek play. And even solitary games yield more than individual experience. We are innately social creatures and we all seek to find greater meaning through community. Endeavor to make your game a nexus of expression for your players. Seek out meaningful ways in which to engage with your community. We need only a moment to think back to a time when a sense of community held a lasting impact on our growth as individuals.
Clique Games is a VR publisher & developer that seeks to push Virtual Reality to new heights. Whether it be Hand-Tracking technologies or advanced physics simulation, Clique Games always reaches higher to set a new bar for what VR gaming can be. Clique Games are high octane, infinitely replayable, and accessible for all players.
If you think we can help you with your VR experience, reach out to us and we’d love to have a conversation.