Matrix level immersive virtual reality?

In the 1999 cult science fiction movie, The Matrix, humans live their lives in a computer-generated world they believe to be real. Their real physical bodies lay submerged unconscious within a capsule filled with nutrient rich fluid while their nervous systems are plugged into a complex AI computer simulation called The Matrix. The simulation feeds their nervous systems with all that is needed for them to believe they are in the real world, living real lives. The premise of the plot is realistic as we indeed create our reality based on stimuli interpreted by our nervous system¹.

Where do we stand today?

We are quite a distance away from a Matrix level Human Computer Interaction but there are several developments that will eventually enable such alternate realities.

Two high-level goals to achieve complete immersion are: Realistically simulate the real world; and immerse humans seamlessly into that simulated reality. The 1st goal is at a very advanced state of development. In today’s games, we are able to roam around in vast realistic computer-generated worlds² and interact with objects in these. The level of realism is only limited by the computing power available. And that is increasing exponentially.

Simplistically speaking, our realities are just real-world data absorbed by our five senses, analyzed and interpreted by the brain as experiences¹. Many of these experiences are stored as memories to be later retrieved and relived. We just need to replicate this technologically. If it’s that simple, we should be able to easily replicate it, right? It appears simple but simple is not easy. Let us see where we stand today.

Data from our realities such as visuals, sounds, smells, tastes, and tactile sensations, are captured, processed/synthesized and stored externally (e.g. a painting, a book, a video, an audio file, a perfume, a synthetic food flavoring, a sculpture, a fabric). And as we retrieve and consume them, we let the brain immerse in these realities again. This works to an extent. We are able to relive portions of our childhood seeing photos. We are able to taste oranges from a synthetically flavored bottled drink. We are able to experience the splendors and horrors of the Jurassic era by watching a movie screen.

But these experiences are never as good as the real thing because the interfaces we have with these external devices are clearly external to us. We are able to clearly differentiate reality from the simulation. We are not immersed in it as we are in our real-life experiences. Enter Immersive Virtual Reality.

Immersive Virtual Reality

Virtual reality has the potential to blur the barriers in Human Computer Interaction to unprecedented levels. It is still a nascent technology and holds much promise. Let's see where VR tech stands now(or in the near future).

Virtual Reality headsets covering most of our field of vision³ simulate immersive visual experiences. Integrated audio systems give realistic spatial audio⁴. Advanced headsets even have odor generators⁵ that simulate smells.

Haptic suits⁶ cover our body generating vibrations and pressure to simulate touch and physical impact.

Research is underway to simulate taste using thermal taste simulation⁷. Applying heat to the taste receptors mimics taste. And electrodes on the masseter jaw muscle simulate food texture and bulk⁷.

Seats and enclosures shake and move in tandem with the action on screen to simulate atmospheres and movement.

Combine these technologies and we are better immersed in a simulation. The brain still doesn’t experience a complete suspension of disbelief. One of the reasons for this is that hardware isn’t advanced enough and the brain is able to perceive the artificiality of the simulation. That is changing rapidly with major global tech players⁸ like Google, Apple, Facebook, Samsung along with newcomers like Magic leap investing heavily in immersive virtual reality hardware, software and content development. Soon we’ll have hardware advanced enough to give us high-quality immersion.

But the fact remains that even with the most advanced hardware, there will still be a barrier between our sense organs and the devices that are feeding simulations to these sense organs. What if we are able to bypass the sense organs completely and feed the simulations directly to the nerves or even the brain? Enter Neural Implants.

Neural Implants

The healthcare industry has been making significant progress in neural implants to help disabled regain some of their lost abilities.

Visual cortical implants⁹ for the visually impaired bypass dysfunctional eyes and feed inputs directly to the visual cortex of the brain.

Auditory brain-stem implants¹⁰ for the deaf bypass the cochlear structures in the ears and send impulses directly to the brain stem.

Sensory cortex implants¹¹ for the paralyzed to regain the sense of touch.

There hasn’t been much medical research on implants for smell and taste even though there are sufficiently detailed brain maps available for these senses. This is simply because there aren’t many of us with serious medical conditions depriving us of these senses to warrant such implant research. But strong demand from the gaming/entertainment industry could change all that.

All these medical implants focus on capturing and delivering real-world data to the brain with the intention of making the disabled experience the real world. What if these implants are fed with simulations instead?

A future of possibilities

As simulations become more intricate and brain implants more refined, we will have a convergence of these two. We will have intricately simulated experiences being directly fed to the nervous system. The brain will finally get suspended in the simulation and believe it to be the actual reality. As a matter of fact, even without brain implants, we already see suspension of disbelief happening with rudimentary VR experiences. Our brains are quite easy to fool.

The biggest barrier to this will not be scientific or technological but social/ethical. Would we as a society let this happen? There will be resistance and apprehension but progress will keep happening independently in related technologies and the convergence will sneak into existence without explicit permission from the resistance.

Let us discuss the ethics of immersive virtual reality in a future post.

Footnotes

Here are some references for a few of the points mentioned in the article:

1. How we make sense of our realities:

2. Photorealistic CGI:

3. Large FOV VR headsets:

4. 3D Spatial audio in VR:

5. Odor generator and haptics for face:

6. Full body haptic suits for VR

7. Taste and chew in virtual reality:

8. The Virtual Reality Ecosystem:

9. Bionic eyes:

A bionic eye that could potentially cure blindness in 15,000 people in the US

10. Auditory brain-stem implants:

Auditory Brain-stem Implant Allows deaf child, Colton, to hear

11. Sensory cortex implants:

Brain Implants Allow Paralyzed Man to ‘Feel’ Through Robotic Arm