VR in a nutshell

2016 has been described as the “year of VR” - but what does that mean?

Image: key concepts for VR in 2016

In a nutshell, it means that a new generation of consumer Virtual Reality (VR) products and services are being launched. The concept of VR has been around for decades but the feasibility of designing, building and staging mass market experiences has only left the labs in the last couple of years.

The origins of this VR wave can be traced back to 2012 when the Oculus Rift Kickstarter campaign raised nearly US$2.5M (947% of their original funding goal). This crowdfunding investment was then dwarfed by Facebook's acquisition of the start-up for US$2B. Oculus is now just one of several high profile brands carving out this new territory.

VR and 360° Video are Not the Same Thing

Before we delve into the types of experience, the first thing to note is that there is a subtle but important distinction between “real VR” and experiences that are in fact just video being played back.

The key distinction is whether the content is pre-rendered or not - that is, whether the content is being created for each individual viewer or is it always the same. Do you ever see anything that the real (or virtual) camera did not capture in the shot? e.g. The viewer of pre-rendered content cannot move their view to look behind a tree. “Real VR” however implies a 3D virtual world that the viewer can freely navigate, and even potentially interact with, “on-the-fly”, that is in “real time”. This means each viewer is “making up” their own unique experience and each time they visit the experience they can navigate a different path through the content. What does this all mean?

If you don't play games, think about your favourite TV show. A 360° video version of that show could allow you to move your head to look at a scene from any angle but apart from moving your head, the experience is the same for every viewer. Everyone gets the same content. The experience may feel more immersive but you can’t move freely in any direction. The camera position is fixed. At the end of the day, it doesn’t matter whether the content is live action, CGI or a mixture of both. If an experience you want to build only requires that the audience view the content from one fixed position, and there is no interactivity, then a pre-rendered video may be all that you need produce. There are a plethora of generic 360° video players on the market including cross-platform ones like YouTube and Kolor Eyes. A host of “VR film” startups also have apps available as a delivery channel for their content e.g. Jaunt, VRSE.

If you've played 3D video games, “pre-rendered” is the difference between watching a video of the game being played (a screencast) and actually playing the game. Watching people play League of Legends on Twitch.tv, a streaming video network dedicated to gamer screencasts, is not the same experience as playing the game.

Technically the only 3D virtual world required for a 360° video is a single sphere with the viewer’s camera perspective at the centre of the sphere.

At the other end of the spectrum, video game developers and media artists are pushing the limits of what can be done in virtual worlds which can be called “real VR”. VR is already taking many shapes, not just in terms of the visuals but in terms of interactive engagement and comfort.

Comfort

You'll be hearing a lot more about comfort in connection with VR. For most people this is not a comfortable medium and VR titles are being rated on their comfort level. What does this mean? There are tactile constraints, such as having to wear a sweaty Head-Mounted Display (HMD), psychological discomfort about looking dorky, and neuro-physiological factors that significantly affect the comfort of content-viewing. Early on, developers discovered that there are limits to how dramatically the viewer's point of view can be altered before nausea kicks in. To limit virtual reality sickness, many titles creatively constrain the user (e.g. you are sitting in a cockpit or behind the wheel within the experience). As techniques mature, the range of movement will increase - but buyer beware.

Comfort level can be used as a creative tool. VR action and horror titles have the capability to shock the senses well beyond traditional screen formats. There have been discussions around new ratings and guidelines. How much discomfort should users be willing to put up with for the sake of immersion? It is early days and everyone is learning as they go. What's clear about the VR space is that big cash is being spent by companies such as Facebook, Sony, Samsung and Google so rely on them to keep the current momentum building for well into 2016 regardless of what actual mainstream appeal there is.

One litmus test for VR comfort is to look at the behavior of the VR makers themselves. As one commentator pointed out recently, no one bothers using the straps. We raise the HMD to our eyes, look around and put it back down. This will change but for now, bite-sized VR pieces are the way to go.

Ten Types of VR

There are, broadly speaking, ten different experiences within three tiers of VR devices currently on offer. Unhelpfully they all tend to be referred to as the same thing. In an effort to break down the catch-all phrase, “VR” into something more meaningful here is a list of 10 types of experience. Let’s step through them to make some sense of it all.


Image: Google Cardboard (lens only), Samsung Gear VR (lens/focusing/orientation tracking), Oculus Rift (lens/focusing/orientation and position tracking)

1) mono screen (no head tracking)

2) stereoscopic 3D screen (no head tracking)

Image: 2D or 3D content viewed through a VR headset

Monoscopic refers to an image recorded from a single viewpoint, so the viewer sees a flat image. Stereoscopic refers to capturing an image from two camera viewpoints, mimicking the view from left and right eye, which creates or enhances the illusion of depth or three dimensions (3D).

The simplest experiences, which you may not consider VR at all, will let you view still images or video up close like with a View-Master from the 1950s (not to be confused with the new Google/Mattel View-Master VR device just released). Forget head tracking, this is simply like strapping a screen to your head. It’s basic but still handy in some contexts. People using HMDs inside a moving vehicle (e.g. watching a movie on a plane) need to be able to turn off head tracking and keep the screen fixed in front of them - especially if the vehicle changes direction sharply. You don’t want content moving in time with the vehicle. The Oculus Videos app for Galaxy Gear VR has a “aeroplane mode” for this purpose, which to-date is still buggy..

3) 360° panoramic stills (head tracking only)

4) 360° stereoscopic 3D panoramic stills (head tracking only)

E.g. sky panorama

The next level up is where a 360° view comes into play. A smartphone, with its built-in cheap accelerometer technology, inserted into a low cost HMD such as a Google cardboard or cheap plastic viewer, can make you feel as though you are peering into a snapshot of a frozen world. The Google Cardboard viewer can be bought off ebay for around $20 and works with either Android or iPhone smartphones.

Image: Google Cardboard v2 (2015), Image3D (founded 1997)

The head tracking in a low-end system only extends to orientation - which direction you are facing. 360 panoramas are nothing new and have been popularised by services like Google Street View but a panorama when viewed through a HMD can take on new life. Flickr and Facebook offer 360° photo experiences.

Technically speaking this is an equirectangular projection of imagery onto a sphere with the user perspective at the centre of the sphere. The projection may also be referred to as spherical or latlong.

5) 360° video (head orientation tracking only)

E.g. cityscape

The remapping of flat imagery onto a sphere is exactly the same technique behind 360o videos. You can look in any direction and see 2D video played back. You can't move the position of the camera.

The capability to shoot 360° video is exploding. For as little as AU$500 you can now buy a small 360° video camera like the Ricoh Theta S that not only shoots panoramic stills but records 360 video (technically as two slightly overlapping fisheye videos). Little handhelds like the Theta S bridge the gap between 360 video capture being a novelty (remember the Flip?) and a future standard feature of every smartphone.

Ricoh Theta S (2015) and CIsco Flip (2010, discontinued)

6) 360° stereoscopic 3D video (head orientation tracking only)

Image: mono 360 stills or video vs stereoscopic 360 stills or video - no head position tracking

 

To get a 360° 3D video requires a lot more effort than a 2D video, and a few technical cheats, because of the complexities in compositing correctly for each of your eyes to see - think layering and stitching images.

Even if you can't move the camera position, a well made 360° 3D video has to make sensible assumptions about where your eyes are in relation to the virtual world (e.g. your left eye is to the left of your right eye). Don’t expect to be able to turn your head sideways, look up and see a strong 3D effect above you. A well crafted experience will typically fade out the 3D effect at the poles (straight up and straight down) to avoid giving the viewer a headache if they attempt that!

Technically speaking what we have here is a stereoscopic equirectangular projection onto a sphere.

New specialist rigs are appearing all the time to facilitate the process of creating 360 video.   Google Jump allows live action 3D content to be captured from a ring of 16 cameras facing outwards. Converting any 3D CGI to a 360 3D video is also a relatively simple process.

On the vendor front, there is a battle underway to own the VR experience. Don’t be surprised if functionality is not available across all platforms. For example while you can play 360° 3D video on most modern smartphones, the app provided on Google subsidiary, Youtube only provides functionality for Android phones via Google Cardboard. If you need 3D 360 on an iOS device, try an app like Kolor Eyes.

7) VR - real-time stereoscopic 3D (head orientation tracking only)

E.g. lookat example

When viewed from one fixed camera position, there is no difference between a stereoscopic 3D video, as described above, and a real-time 3D view but the moment when you need to move the camera or modify the experience for different users (e.g. for a game) then you are entering the world of real-time 3D graphics. The skillsets required to produce 360 video (e.g. live action filming, compositing) are not necessarily the same as what is required for  real-time 3D (e.g. 3D modelling and animation, software development) so it is important to know what you’re talking about.

The first of the commercially released headsets (this generation) was Samsung’s Galaxy Gear VR. Deceptively simple (no cables attached), the Gear VR is arguably the most accessible VR experience at the moment with great head-orientation-only tracking. A more comfortable headset, a powerful focusing system (for even my -5.25 eyesight!) and some built-in head tracking makes for a much more seamless experience than a Google Cardboard. That said, the lack of head position tracking could be a deal-breaker. No walking around with this one on.

Another consideration with real-time VR is the graphics processing capabilities of your device. 360 video is surprisingly easy for recent smartphones to run but when it comes to real-time rendering, there are limits to what a smartphone can do. Both Oculus and HTV have release “standard VR specifications”  to help people determine whether they will need to upgrade their computer gear. A new graphics card may be on the cards.

8) VR - real-time stereoscopic 3D (head orientation + position tracking)

Image: real-time VR with and without position tracking

 

When the position of the person’s head can be tracked (not just the orientation) then the experience has the potential to become a whole lot more realistic. Oculus Rift, HTC Vive and Playstation VR all provide some position tracking but at the expense of tethering the user, with leads, to a more powerful computer. The amount of tracking data required to make all of this work is substantially more than what is required for tracking head orientation so a smartphone isn’t going to cut it. These upper-tier HMDs are nothing like the clunky devices from the 80s (when the sheer weight of the contraptions were sometimes more than your neck could bear) but it is worth noting that some people will prefer a less immersive experience without any wires and cables.

The virtual world you experience from these devices relies on a complex interplay between the viewing system (e.g. a Head Mounted Display), position tracking cameras, and high performance audiovisual software (often referred to as "the graphics engine") that is capable of drawing (rendering) the details of the world fast enough to keep up with your exploration.

Moving through highly detailed 3D environments and viewing characters from any angle will showcase this flexibility.

8i has showcased a process under development for converting video footage from an array of inward-facing cameras into a photo-realistic 3D avatar.


9) VR - real-time stereoscopic 3D with head + controller tracking

All the top tier HMDs support hand-held controllers that are tracked along with your head. Bringing your hands into play, using conveniently placed buttons, widens the possibility space for VR creativity.

 

Image: hand-held controllers come with the top-tier VR headsets

This is where things start to get really interesting. All the top tier HMDs allow for some degree of physical movement but the area that can be tracked varies wildly. The HTC Vive system stands out at the moment by covering a 5 x 5 meter area.

10) VR - real-time stereoscopic 3D full body tracking

Extending head tracking to also tracking hand-held controls and even the full body (like what happens with motion capture rigs) then you have what you could consider to be true VR.


Image: motion capture systems coupled with VR - the skies the limit

Couple this with the ability to track the body (from eye movement to full body skeleton tracking), and you have a pretty good idea of the canvas VR creators will be playing with over the next couple of years. Look out!

 

HMDs in a nutshell

High-end:    HTC Vive, Oculus Rift, Sony Playstation VR

Mid-tier:     Samsung Galaxy Gear VR (requires Samsung smartphone)

Low-end:     Google Cardboard (requires Android or iOS smartphone)

VR engines in a nutshell

High end:     Unity, Touch Designer, Unreal

Mid-range:     jMonkeyEngine

Low-end:     aframe (WebGL)

Conclusions

It is fair to say that this is the year of VR, and that there has never been a better time to check out the possibilities. We are witnessing a new digital frontier that shows no signs of settling down. With billions of dollars being pumped into nascent VR brands, there is a lot riding on the next couple of years development. Punters looking at the substantial costs of being an early adopter can afford to sit back and wait for prices to come down. VR sceptics point to 3D and 4K TV as an illustration of how the current hype around VR could soon fade. Nonetheless VR has been evolving for a long time and today's excitement reflects iterative improvements made after decades of hard research and development.

As someone who began experimenting with Virtual Reality Modelling Language in the mid-90s I am both excited and sceptical about VR's potential in the short-term.  

What really piques my interest is the storytelling potential. For storytellers the field is wide open. The rules of narrative and storyworld design for VR are not clear-cut.  The VR story is not just about consumer propositions on offer. A mainstream VR experience platform will be as transformative a step as the original World Wide Web was. But why are we doing this? If Wikipedia can be described as a cathedral of the modern age, I’m curious to see what VR constructions will take hold of popular imagination. My interest in the VR scene is as much about the personalities and organisations behind the scene as the tech. We have this relentless drive to develop new digital futures regardless of the state of the real world. Hopefully VR development will reflect diversity in the way the open web has.

Credits

Thanks to all the VR pioneers, inventors and early adopters who’ve laid the groundwork for this article.

@michela

Last updated 20160329