Sunday, December 14, 2014

Why is fingertracking a must for the future of VR

Virtual Reality allows us to see new worlds as if we were inside of them, but without ability to touch and interact with them, it'll never be complete. I wrote down some of my thoughts on finger tracking for virtual reality.


I love making content for virtual reality and showing them to other people, observing their reactions as they suddenly find themselves in the worlds that I have imagined, designed and programmed to respond and react in the ways the real world wouldn't.

One of the most common reactions to first putting the Oculus Rift on is putting the hands in front of one's eyes and asking why they can't see them. Right now, you're invisible in many of the VR experiences. Sure, you can look down and see a body, but it doesn't do anything that your body does, it's not really yours. The easy way around this is to position yourself in the same way as your VR counterpart to increase immersion.

But doing VR right isn't going to be easy. The head mounted display takes care only of the visual part, but there's a lot more to the world than just seeing. We have a very long road ahead to cover all the other senses, but even now some technologies are exciting and quite useful for VR, namely finger tracking.

Technologies like Leap Motion, ControlVR or Nimble Sense resolve the common question “Where are my hands?!”. Sure you could've used a Hydra/STEM, but I don't grab some object and move it in front of my face to see my hand... that's what I do when I want to see an object I've grabbed, which is what devices like STEM are good for – holding things in VR, like mech control handles or guns.

Don't get me wrong, they have amazing applications as well and I have great plans for such controllers in VR, involving some over-the-top fast-paced action with unique hi-tech virtual “guns” that make you feel like you're holding a portable nuclear bomb. Using them for various precise tool tracking applications, such as 3D drawing or modelling, makes perfect sense as well, since you're actually holding some tool, instead of using your bare hands.

However what I want is to see my hands as I would see them in real reality (can we call it RR? :3). But what's more, I want to use my fingers – and this is where devices like STEM are coming short. They can track overall position of my hand, but not my fingers – it's effectively like having stubs in place of your hand.


Very early version of World of Comenius using the Hydra controllers.

When Leap Motion introduced the VR tracking, I quickly glued my Leap Motion to the Oculus and started playing around and integrating it with my own applications, namely World of Comenius. At the time, it was sadly glitching to the point of being unusable. Luckily people from the Leap were very helpful and listened to the feedback and criticism and over the time improved the tracking software significantly and it's likely to continue on that road.


Early version of the Leap Motion integration, with quite painful glitches that made it mostly unusable.

Fingers are extremely versatile appendages. Just think about how many things we do with them. We write, we paint, we type, play games, interact with variety of devices or even gesticulate. The article you’re just reading is result of series of fine movements of my own fingers and so was the virtual reality software I have created.
That software is built on top of work of a lot of other people, from Leap Motion, Oculus or Unity, running on hardware designed by thousands of engineers, inputted into computers with their own fingers. Much of the technology, software, art and entertainment in our society was expressed and built by a long series of fine finger motions.
Interacting with virtual reality using your fingers is therefore crucial, as it opens a plethora of options. If you want to grab an object in front of you... well you go and grab it. Sounds better than moving a controller thingy near it and pressing and holding a button to express such action. No, not that one! Move your finger higher. No not that high, ugh let me just position it for you since you can't see through the Oculus...

Of course things are still far from perfect now. Not only the finger tracking solutions glitchy, high latency or expensive, but we are completely missing tactile sensation. We can interact with the world, but we can't feel it yet. It's like interacting with intangible holograms – which is the narrative I've actually chosen for World of Comenius, including a hologram-intersection effect when you put your virtual hand through the virtual object.


World of Comenius in its current stage, Leap Motion tracking improvements made it quite usable in most cases.
Fortunately there are already solutions appearing, such as using ultrasound array to give a tingly feeling to our fingers in the midair. Now we've got some semi-tangible holograms that we can interact with, which is a lot better.
But even with mid-air haptics there's an important element missing: Our sense of proprioception. When the fingers get near an object, sure it starts tingling, but we can still put our hand through it. We can't press the fingers against its surface, or grab it with your fingers and have the sense of not being able to push through and our fingers staying in the same position, despite the pressure we're exerting.
We're not interacting with tangible holograms, but still holograms. I'm not sure what the solution to proprioception is going to be. Maybe some simple one could involve some pads on fast robotic arms with the finger tracking, quickly moving near anything that the fingers can touch and exerting a force against?
I don't know, but I certainly hope it will get resolved before we go full VR and just directly stimulate our brains and read their outputs – because that's probably going to take a while.
Meanwhile, we'll have to make do with what we have. It may be glitchy and far from perfect, but it's still fun to play with. No technology was ever perfect from the beginning and ambitious goals are going to take a lot of work to achieve, so we can at least enjoy the increasingly more convincing glimpses of the perfect VR as we progress forward.

Coming next: Behind the scenes of SightLine.

7 comments:

  1. "But even with mid-air haptics there's an important element missing: Our sense of proprioception. When the fingers get near an object, sure it starts tingling, but we can still put our hand through it."

    This may not be an issue. Remember when smart phones became popular, there was no tactile buttons for the app icons and various interactions. Now a few years later we sort of don't care any more, we adapted to the lack of a 3d physical button and got use to it. The same may happen in 3D space. Instead of relying on touching and compression of our finger agasint a surface, we are now free to move our hands in 3D space and feel sensations along it. Maybe the keyboard of the future will feel like fuzzy channels we stir about in space :P

    ReplyDelete
    Replies
    1. I don't think getting used to VR being less real to the point that it becomes a non-issue is the right approach - I'd definitely like these problems to be eventually solved, because it expands you options.

      The advantages of the touchscreen outweigh the lack of tactile sensation, but that hardly means that they are ideal - it's difficult to hit the right element, so the UI's have all become huge so you can be really sloppy with them, you can't type as fast on touchscreen as on a real keyboard and generally there are many limitations. Funnily there are appearing solutions for this as well - technology that dynamically creates bumps on the touchscreen display:
      http://tactustechnology.com/

      Delete
  2. Great article! However, I need to point something out: Proprioception is the internal sensing of the positions of your own limbs. Proprioception does not need to be "solved", since it's there all the time, in VR and RR. What you are referring to as "proprioception" is simply the sense of touch.

    ReplyDelete
    Replies
    1. Yes, I know what proprioception is and it definitely needs solving, because I am not talking about sense of touch. Say you have gloves that emulate sense of touch.

      You start touching some virtual object and you feel it, but then you push harder and your hand just goes through, even though the object has nowhere to go - your arm changes position in space. Sure it feels pressure, but because of the proprioception, you can tell your hand is in different place than it should be (like you pushed your hand through a wall).

      If we didn't have proprioception (like the octopus for example), then we'd have one less problem to deal with.

      Delete
  3. Not sure if solving proprioception is the right way to do it. What would that mean? Would you make something so you believe your arm is somewhere else than it really is? If you did, you still wouldn't feel increasing pressure when pushing against something, like when you're pushing against a wall.

    I do believe some kind of physical barrier is needed. I've tried something like CyberForce before and it works pretty well.
    http://www.cyberglovesystems.com/products/cyberforce/overview
    http://www.cyberglovesystems.com/products/haptic-workstation/overview

    Not sure if something like CyberForce or their Haptic Workstation is the future though, as its a big big and clunky. For gloves perhaps gloves where you are moving air or liquid around filling and emptying some bubbles, or maybe something completely different.

    If not some kind of physical barrier, then perhaps it might be possible to block the signal to the muscles and send new proprioceptive signals to simulate VR objects. That is closer to a neural interface, but you don't have to plug something into the brain, maybe you could stimulate some of the nerves in the muscles with electric field or something like that.

    ReplyDelete
  4. There are some applications where haptic feedback in VR could be convincingly accomplished, for example in flight simulations. A real world cockpit mock-up can be built where the position and haptic quality of instruments, dials etc is accurate, but that does not look like the real thing and does not have real functionality other than providing positional haptic feedback. This would be possible in all VR applications where the position of the operator is in a fixed environment. To make the VR experience convincing, a real time VR body should show the same position as the operator's fingers/hands/body, established by a tracking system.

    ReplyDelete
  5. http://bantal-silikon15.blogspot.com/2015/05/welcome-to-wordpress.html
    http://bumbu-pecel01.blogspot.com/2015/05/a.html
    http://seo-guru01.blogspot.com/2015/05/asf.html
    http://seo-konsultasi-mudah.blogspot.com/2015/05/fgre.html
    http://marinir-seo-konsultan-12a.blogspot.com/2015/05/asf.html
    http://dfes12.blogspot.com/2015/05/gre.html
    http://agen-seo19.blogspot.com/2015/05/wer.html
    http://agen-seo20.blogspot.com/2015/05/as.html
    http://agen-seo16.blogspot.com/2015/05/er.html
    http://agen-seo18.blogspot.com/2015/05/rqw.html
    http://jksbindo.blogspot.com/2015/05/rte.html
    http://jkt-49.blogspot.com/2015/05/tyu.html
    http://seokonsultan1.blogspot.com/2015/05/asrw.html
    http://guru-seo-malang.blogspot.com/2015/05/arte.html
    http://seo-aaa.blogspot.com/2015/05/rty.html
    http://marinir-seo-8.blogspot.com/2015/05/rty.html
    http://vvtiin.blogspot.com/2015/05/rqw.html
    http://agen-seo17.blogspot.com/2015/05/rtq.html
    http://asdf-14.blogspot.com/2015/05/qwr.html
    http://bantaljkt1.blogspot.com/2015/05/qwa.html
    http://larismanisyaluk.blogspot.com/2015/05/q.html
    http://bantalku1.blogspot.com/2015/05/b.html
    http://bantal999.blogspot.com/2015/05/r.html

    http://distrobantal.blogspot.com/
    http://bantalsilikon01.blogspot.com/
    http://sabunherbalya.blogspot.com/
    http://kesethandukmalang.blogspot.com/
    http://bantalmalangmurah.blogspot.com/
    http://restoran-seo.blogspot.com/
    http://distro-seo.blogspot.com/
    http://jualsangkarpleci.blogspot.com/
    http://marinir-seo.blogspot.com/
    http://kopiluwakliar01.blogspot.com/
    http://bumbu-pecel-malang.blogspot.com/

    ReplyDelete