A new version of RUIS is available at our download page. It finally adds support for Oculus Rift CV1 and HTC Vive, while requiring Unity 5.4. All head-mounted displays and motion controllers (including Oculus Touch) that support OpenVR can now be used in RUIS. They can also be calibrated to operate in the same coordinate system with Kinect.
Choices for calibrating device pairs
A good example of using Kinect body-tracking together with a head-mounted display in the same coordinate system is demonstrated by our Vertigo demo (which now also supports HTC Vive):
This is the first time since the Oculus Rift DK1 and its extended mode that it is again possible with RUIS for Unity to render simultaneously to a head-mounted display and other displays, including CAVE setups:
4-wall CAVE test (panorama)
If you’re rendering on other displays besides head-mounted displays in RUIS, you should be aware that Unity has a bug with custom projection matrices, which messes up shadows. Unity 5.5 will apparently get rid of the bug. Until then, there is a quick fix available however.
See the latest RUIS readme for more information about this release.
The next version of RUIS for Unity will include a simple OpenVR calibration process that allows you to match the coordinate systems of an OpenVR device and a custom tracking system of your choice (e.g. OptiTrack mocap setup). My aim is to enable developers to easily use head-mounted displays with their own (possibly high-end) full-body mocap systems, thus enabling simplified development of first-person avatar experiences.
I recently gave a talk titled “Staying Ahead of the Curve in Virtual Reality” at ARTtech Seminar of Assembly computer festival (video embedded at the end of this post). In the talk I discussed some of my own work, and gave an overview on current consumer VR and its near future. In this post I want to focus on the following three considerations for VR developers that I laid out in the finale of the talk:
1. Justification for VR
Ask yourself, why does your application need to utilize VR? Are you using VR just as a gimmick? Games and other applications do not get magically better just by converting them into VR. The use of VR is a no-brainer in certain types of games and entertainment where additional immersion and depth cues are important. But that is not the case for all games (2D games being the most obvious ones), let alone business software.
Take into account that currently the added immersion of head-mounted displays come at the cost of ergonomics; the user has to wear a heavy, sweaty headset that induces vergence-accommodation conflict and suffers from a low resolution. Over time all these issues will be alleviated and even removed altogether, but people will still be using 2D displays far into the foreseeable future. The added value from the use of VR must outweigh its costs for your application.
Furthermore, VR interfaces and traditional interfaces excel at different tasks (I’ll discuss that in a future post), and it is not clear yet in which application domains the use of VR brings noticeable benefits. Entertainment and training VR applications seem like safe bets. But the net benefits are less clear for other domains. For example, in the domain of 3D content creation we can pose the following question: which of the numerous real-world 3D modeling and animation tasks faced daily by production companies become more efficient in VR? Further experimentation is needed from researchers and practitioners of VR.
2. Social VR
I encourage VR developers to give thought on how they could benefit from having people interact together within their VR application. Mere head and controller tracking adds a good amount of expressive power to non-verbal communication in VR. A glimpse of this can be seen at the end of the Oculus Toybox demo video. Humans are social by nature, and all the major consumer VR companies are looking into bringing more human expressions into VR. Thus, in the future consumer VR tracking systems will track our whole bodies, eyes, and facial expressions.
At one end of the social VR spectrum is the concept of metaverse, along with massive online virtual worlds and social networks. AltspaceVR and JanusVR are examples of early attempts at creating immersive, online virtual worlds. At the moment they are still relatively small scale and could be described as VR playgrounds for multiple users. In contrast to open virtual worlds and VR town squares, there will also be demand for more intimate telepresence applications for two people or small groups to convene. Such applications could be a part of a larger metaverse or made available as a standalone, single-purpose app like Skype.
VR game developers should consider adding social dimensions to their games, even for single-player experiences. Simplest way to do this is to implement spectator modes and live VR streaming. A company called Vreal is developing a software platform for that purpose, with the aim of becoming the Twitch of VR.
Multiplayer games that involve waiting (e.g. for the start of the next match) could include simple VR waiting rooms for social interaction. As online communities tend to be riddled with abuse, you shoul look into ways of reducing such bad behavior.
And lets not forget the possibilities that online collaboration in VR presents for business software, as demonstrated by MiddleVR‘s Improov3 demo:
3. Augmented Virtuality / Multitasking
Many current consumer VR headsets completely block the view to the real world, requiring the user’s full attention. If you’re a VR game developer, ask yourself whether your game is captivating enough to keep players away from other activities. Because when the player gets bored, the only escape is to remove the headset and quit the game.
All games become repetitive sooner or later, when the player gets used to the gameplay mechanic or is forced to do mindless grinding in order to proceed. Nevertheless, many players keep playing because they want to see what the game has to offer around the next corner. In non-VR games the players can cope with the boredom by simultaneously engaging some additional activity.
According to a study by Foehr (2006), American teenagers frequently consumed another media while playing video games; 41% of the total time spent on video games involved such multitasking. Playing video games was most commonly paired with watching TV and listening to music. Due to the reality-occluding nature of head-mounted displays, VR developers need to see extra effort to provide similar multitasking capacities that are implicitly present in non-VR games.
HTC and Valve were smart to allow some real-world multitasking by equipping the Vive headset with a camera. This augmented virtuality implementation allows the user to utilize their keyboard and grab their drink while wearing the headset.
Fortunately VR can be augmented with anything, and we are not limited only to real world elements. Envelop VR and Virtual Desktop applications already demonstrate how to use 2D applications in VR. But rather than have a VR desktop application, I’m more enthusiastic about bringing arbitrary 2D content and software streams inside any VR application or game. This will allow the user to seamlessly multitask between the 2D streams and the actual VR application where they appear. I created the below concept image using a screenshot from Pool Nation VR to illustrate this idea:
In this example the player has augmented the game with two additional 2D streams positioned in the virtual world: a Reddit browser tab and an episode of The Office (streaming from a hard-drive or online video service such as Netflix). Both streams could be viewed and interacted with while playing virtual pool.
VR developers could allow user-specified 2D streams into their application: video screens, browser tabs, and 2D applications. This is somewhat analogous to custom radio stations of GTA games, which allowed players to listen to their own music while playing. Of course it doesn’t make sense for every developer to implement their own 2D stream functionality, and a Unity/Unreal plugin with those capabilities will emerge in the near future.
To summarize this section: a VR developer should decide if their application allows “in-app multitasking” or requires full attention from users. The latter approach could be too much to ask as far as games are concerned.
There are a number of compelling reasons about why now is a good time to start working with virtual reality (VR). Most of these points apply to augmented and mixed reality as well.
I start off by introducing my two main assertions regarding the current situation of consumer VR:
1. Anyone can become a VR pioneer. Consumer VR is being invented right now, and VR as a media is being defined. VR research has been conducted for decades and in that regard experienced VR researchers have an advantage over hobbyists. But the spread of consumer VR hardware leads to democratization of this new media; now the masses can also have access to high-performance VR equipment. This situation can be compared to the arrival of home video cameras, which gave rise to influential film makers such as Stephen Spielberg.
2. Individuals and small teams can make a big impact in VR. Many areas of consumer VR are uncharted territory, and in that sense independent developers and big companies are on the same level. Both can make major contributions in the field. You could for example get lucky and create a killer application for VR; think of Minecraft, Kerbal Space Program, and other small projects that came to be very influential. Not all discoveries and innovations need an enormous budget and a large team of engineers. A few years from now it will be more difficult to influence the field of VR as it becomes more established. *
This ongoing pioneering phase of consumer VR offers numerous opportunities for developers and creatives:
Finding application domains and niches where the use of VR is advantageous.
Developing tools for VR content production and distribution, e.g. the photoshops and wordpresses of VR.
Pioneering VR content fundamentals, such as effective VR storytelling techniques.
Creating better 3D user interfaces for VR and helping to define related standards.
Engineering hardware innovations.
Some of these opportunities will have big monetary rewards, and some have important consequences for the future of human culture and the way we interact with computers.
The hype surrounding VR is currently helping developers, and makes it easier to pitch your ideas. But be aware of the inflated expectations that will inevitably lead to a backlash of some magnitude. Furthermore, a mass adoption of consumer VR devices takes several years. So prepare for the long haul. Unity co-founder David Helgason suggested that new VR companies should have funding for the next 2-4 years.
Most developers in the Western world will target the English speaking market. However, one should not underestimate the Chinese VR market, and the possibilities that it offers. For example, gaming cafes are very popular in China, and they are gradually equipped with VR devices.
And finally, lets not forget that VR is fun and the technology provides countless ways to express yourself, both artistically and experimentally. If you want to get involved with VR development, but don’t know where to start, these answers on Quora will point you in the right direction.
* Assertion 2. relates to user-led innovation, which is a concept where not only big producer companies are a source of product innovation, but also individual users (including small companies) and communities. In my PhD thesis that is currently being reviewed I discuss VR also from a user-led innovation perspective, while the main focus of the thesis is in VR application development. One example of current user-led innovation in the field of VR is the abundant experimentation with different forms of VR locomotion:
In May I had the opportunity to try out Microsoft Hololens. It had a phenomenal inside-out positional tracking, which felt very robust. As reported widely online, its field of view is very limited. That is the single biggest obstacle for usability and immersion. As a self-contained wearable display device, Hololens is a great “development kit” for augmented reality developers to start experimenting with the technology. I believe that it will be useful in a number of real-world cases, despite the narrow field of view.
I was surprised that the interactive cursor was locked to the center of the display, and could only be moved only by rotating my head. I was expecting to be able to relocate the cursor by moving my hand in front of the device, because hand gestures are also used for “clicking” and bringing up the menu. Hololens also comes with a wireless clicker peripheral that can be used instead of the gestures. That would be my preferred way for interaction due to the clicker being more robust and ergonomic. Perhaps the “locked” cursor is a good idea after all, for those same reasons.
Coming RUIS for Unity Update
A few words about the future update of RUIS for Unity: The currently distributed version 1.082 still requires Oculus Runtime 1.06, which is obsolete and does not support Oculus Rift CV1. I have created a beta version of RUIS that supports HTC Vive, which I used for adding Vive support for the Vertigo experience. I have not made that version public, because it is still very much a hack. I’m waiting for Unity to release stable version of Unity 5.4, which eases my job by adding native support to Vive and unifies the head-mounted display interface.
I have submitted my PhD thesis (about virtual reality) for review, but I still have a bunch of other projects I’m working on. Therefore the new RUIS version will probably come out in July or August. It’s worth the wait 🙂
I was going through my old photos and found something that I should have blogged years ago.
You see, I met the ”Godmother of Virtual Reality” Nonny de la Peña at IEEE VR2013 conference, and we talked about her work. At that point I had already heard about her Guantanamo Bay detainee VR experience, where the user has to endure in a stress position while hearing “interrogation noises” from the next room.
What a way to put yourself in another person’s shoes! And that is the idea of immersive journalism, a concept coined by de la Peña. You could watch the news from a small box in your room, or you could experience the news in first person with the aid of virtual reality.
I liked Nonny’s ideas and asked to see more of her work. She was very hospitable and in March 2013 she gave me and my friend a tour at the USC Institute for Creative Technologies (ICT) in Los Angeles.
First Nonny showed us her production Hunger in LA. It’s an immersive journalism piece where the driving force is real audio recorded at Los Angeles food bank. At the time of the recording there were delays in food distribution, it started to get crowded, one person had a seizure, and ambulance had to be called.
Nonny described that some viewers of Hunger in LA had been so touched by the experience that they cried. This didn’t happen with me, or my friend who was a VR novice. Perhaps we were all too jaded for that. But the use of real, non-acted audio was very moving, and I can see how people could have strong reactions for such authentic content.
There were all sorts of set pieces in the ICT laboratory as seen in the background of the above photo. Apparently they have been working closely with US army, exploring military training applications of virtual reality.
We got to see different parts of the laboratory. Above is a lab desk full of prototypes at USC ICT, where Palmer Luckey worked as a lab technician with FOV2GO head-mounted display.
Oculus had given DK1s to ICT before the official shipping date of March 29th 2013.
This was one of the mobile head-mounted display prototypes that we tried.
We also got a chance to see ICT’s Light Stage, which has been used to capture 3D scans of actors for several movies. Many thanks to Nonny de la Peña for giving us the tour and sharing her work!
In November I got to borrow a HTC Vive dev kit for a few days. I was responsible for setting it up at a AEC hackathon in Helsinki. Before I share my experiences in detail, here are my two suggestions to Valve:
Allow developers to opt out from auto-updating SteamVR software. If I have a working demo configuration, I don’t want an automatic update to break it with a new plugin version etc.
Do not require Steam to run in the background of SteamVR. This is just common cleanliness, as it was a bit annoying that every time I started SteamVR it also launched Steam.
Overall the experience, particularly running the demos, was great. I already had tried the Aperture Robot Repair demo in August, but the display quality and tracking accuracy still made me very happy. I got very positive reactions from my colleagues in Aalto University to whom I showed the Vive for the first time.
Getting the dev kit to work took awhile. After connecting and placing the physical hardware, SteamVR wasn’t able to properly access the hardware no matter what I tried. And there were plenty of things to try, as can be seen in the SteamVR forum. It wasn’t until I updated the firmware for the HMD and the controllers that everything started working. I had to use rather unwieldy command line tools, whereas Oculus Rift DK2 had offered a simple firmware update process through the Oculus Configuration Utility.
When all the systems showed green status in SteamVR, creating and running a test VR scene in Unity Editor was a breeze. As a side note I’m happy to see that Unity is integrating VR functionality directly into their engine, which eases development for different VR platforms. I’d imagine that Epic is doing the same with Unreal Engine.
Occasionally the Vive stopped working after restarting my computer, and I needed to uninstall Vive’s USB drivers and reboot to solve the problem. According to Valve this issue is related to Vive’s HMD control box. From what I understand the vast majority of problems reported in SteamVR forum can be attributed to that. I believe that the situation will be much better with the 2nd Vive dev kit.
I didn’t have any problems with the tracking quality and everything ran smoothly. My only grievance is that I couldn’t install any of the cool Vive demos from Steam. Currently Valve has to separately set those privileges for each Steam account, and we couldn’t get them to do that in the short time-frame that we had. Instead I resorted to googling for unofficial, 3rd party Vive demos, which understandably had lower production values. For some reason I needed to run each demo in administrator mode to get them to work.
To summarize: this first HTC Vive dev kit, its hardware and parts of the software, feels like it was hacked together by a group of scientists in a lab. What actually happened is perhaps very close to that. In contrast Oculus DK1 and DK2 were slightly more polished, because as a pioneer Oculus had more to prove. This is not a complaint, and I’m quite happy that HTC and Valve decided to grant developers such an early access. HTC Vive, particularly its Lighthouse tracking, is just so good that it’s easy to overlook the lack of refinement in this early dev kit. I hope that soon I will get a permanent access to HTC Vive, so I can integrate it into my RUIS toolkit, enabling developers to combine room-scale Lighthouse tracking with full-body tracking of Kinect.
An article by researchers from The University of Louisiana at Lafayette reviews RUIS among with two other virtual reality toolkits for Unity. RUIS did very well in the review, and the original version of the article that I read in August stated that
with RUIS being free and highly versatile, it is the clear winner for low budget applications.
The author changed the wording in the final article version to “promising for low budget applications“, because their adviser suggested using a wording that sounds less biased. Oh well 🙂
In the article RUIS reached almost the same score as MiddleVR (a professional $3000 toolkit) which came on top when price was not considered, as seen from the below table adapted from the article:
Performance & reliability
CAVE display flexibility
Ease of use
In the above table each category was given 1-5 points.
In terms of “Documentation and support” getReal3D scored 10, MiddleVR 20, and RUIS 14. Improving RUIS documentation and providing tutorials is in our todo list.
The article is slightly mistaken in that it says that top and bottom CAVE displays are not supported by RUIS, but this is not the case. The display wall center position, normal, and up vectors just need to be configured in RUISDisplay component. Please note that RUIS is mostly intended for CAVEs with a small number of displays because each view is rendered sequentially. For faster CAVE rendering in Unity you should probably try MiddleVR or getReal3D, which offer clustered rendering.
The article was published in International Journal for Innovation Education and Research (IJIR).
P.S. I participated in Burning Man 2015, where I demoed our Vertigo application at VR Camp. They had a dozen of computers with Oculus Rift DK2 and a HTC Vive. Here is a photo of me trying out Tilt Brush on HTC Vive.
Yesterday I took part in FIVR (Finland VR) meeting and got the chance to try out HTC Vive and other VR technology that I haven’t tried before. The FIVR group started meeting autumn 2014 and within one year the number of participants grew from four people to 40. This is mostly thanks to the Finnish game company Mindfield Games who have been very active in organizing the events.
The meeting and our demo was featured in TV news and an online article by Finnish public broadcasting company YLE.
Valve’s Portal VR demo for HTC Vive is the best VR demo that I’ve seen (disclaimer: I haven’t tried Oculus Rift CV1 yet). While interaction-wise it’s nothing special, Valve’s high production values have resulted in an audio-visually beautiful piece in the familiar Aperture Science environment that is loved by many. A big part of the experience is the HTC Vive hardware that performs exceptionally well; The 110 degrees 2160×1200@90Hz HMD takes immersion one step further and I’m yet to see such precise and low-latency 6DOF tracking, all working in a room scale.
Activity Core‘s “Sensored Swissball” was a surprisingly fun control device! It’s perfect for all kinds of, uh, riding interaction.
Virtual Boy by Nintendo! 20 year old VR technology. Wario Land is actually a pretty good game.
We don’t have HTC Vive yet, but hope to get one in the future. We are still working on a Kinect 2 + Oculus Rift demo, and after that we’re planning to modify the RUIS code architecture so that it will be easier to add support for upcoming VR input devices (e.g. Oculus Touch, Sixense STEM).
An upcoming Kinect 2 + Oculus Rift DK2 demo created with RUIS and Unity 5 (no release date yet).
Unity 5 allows the use of Kinect 2 and Razer Hydra without a Pro license, so it makes sense to update your RUIS project to Unity 5. In May June we will put out an official RUIS for Unity 5 release. Meanwhile you can use the below guide to upgrade RUIS to work in Unity 5.
1. Create backup of your project.
2. If you have modified any RUIS scripts, prefabs, or scenes, you need to create duplicates of them because RUIS files with the original names will be overwritten by the RUIS update.
3. Install Unity 5. If you are a Windows user and intend to use Kinect 1, you should install the 32-bit Editor (Additional Downloads, For Windows link): http://unity3d.com/get-unity/download?ref=personal
4. Open your project with Unity 5, and Upgrade the project when Unity asks to do so.
5. When “API Update Required” appears, choose the option “I Made a Backup, Go Ahead!”.
6. The project should be open now. Ignore any errors in Console, and open the RUISunity1071_Unity5.unitypackage file in Explorer/Finder and import everything.
7. After importing, delete the following files and bundles in \Assets\Plugins: KinectForUnity.dll, libOculusPlugin.so, OculusPlugin.dll, sixense.dll, sixense.bundle. Do NOT touch any files in the Android, Metro, OculusPlugin.bundle, x86, and x86_64 subfolders.
8. Everything should work now, unless some of your own scripts or assets are broken.
P.S. If anyone knows a math library for C# that supports (or can be easily modified to support) in-place matrix operations for multiplication, addition, inversion, and transposition of 4-by-4 matrices, let me know! The current matrix library that we are using (and Unity’s Matrix4x4) allocates new memory after every matrix operation, which causes very frequent garbage collection and results in a noticeable performance loss when Kinect rotation smoothing is enabled in RUIS.