Below I have written down those of my ideas that I am pleased with, but which I have chosen not to pursue further. Feel free share your thoughts on the concepts presented here, or utilize them if you are inclined to do so.
Click the titles to see the full description for each concept.
AR headset that enables seamless switching between VR development and testing
Inception date: 23.1.2017
VR developers could wear AR headsets while developing, because wearing them does not prevent the use of a standard development interface (monitor, keyboard, and mouse). When the developed VR application is tested, the AR headset would be used to display the VR content. Thus there would be no need to spend time on putting on a VR headset (or alternatively flipping down its display)...
This reduces the effects of “context switch” that occurs between each iteration of development and testing cycle. This is relevant, because constant testing and re-implementation is the second-most severe issue for VR developers according to the data (ratings from 132 VR developers) collected during my PhD research.
The AR headset would be primarily intended for testing the VR application’s user interface and interactive features. Color balance and other graphics related matters should still be tested with the targeted VR headsets, because optical see-through AR headsets obviously can’t block light, and video see-through has its own issues. Each major consumer VR headset could have a developer-only AR headset equivalent that utilizes the same tracking method, so that testing with the AR headset would be as close as possible to testing with the consumer VR headset.
This idea came to me while I was thinking of new ways to alleviate the issues in VR development in preparation of my PhD defense.
Killdozer, a sitting VR Experience based on a true story
Inception date: ~2015
While pondering about sitting VR and how enclosed virtual spaces seem like a natural fit for such experiences, I happened to remember the story of Colorado resident Marvin Heemeyer and his 2004 Killdozer rampage...
Heemeyer has become an underground hero to some, representing a vigilante who stood up to “the man”. His story contains many thematic elements that would work well in a game: injustice, vigilantism, running amok, and the power fantasy associated with the unstoppable machine, built by one man. In some sense Heeymeyer is a working class’ Tony Stark, Batman, and the Punisher all rolled into one.
As far as game mechanics are concerned, there could be at least two phases: building and designing a Killdozer, and then rampaging with it. The game could be mindless, GTA-style destruction derby, or involve more introspection on what it’s like to be on a last journey in the belly of a beast of your own making. Surreal elements could be easily added to the events of the game.
Heemeyer spent 18 months upgrading his 50 ton bulldozer to a tank; he used concrete and metal to build an impenetrable enclosure equipped with 3 rifles. Upon lowering the protective concrete hull on top of his bulldozer cabin there was no way out; he became one with the Killdozer, video monitors hooked onto external surveillance cameras being his only view outside. Heemeyer then went on a rampage in his hometown, destroying property of those whom he held grudges against: city officials, businesses, and the police.
“Sometimes reasonable men must do unreasonable things.”
– quote from one of the notes left by Marvin Heemeyer.
Nothing could stop the Killdozer’s rampage: the local police, state troopers, and a SWAT team fired the bulldozer over 200 times, planted explosives on it, and dropped a flash-bang grenade in its exhaust pipe. A heavy wheel tractor-scraper was sent to face off the Killdozer, only to be humiliated by it. Colorado Governor even considered authorizing the National Guard to use an Apache attack helicopter to stop the Killdozer with a Hellfire missile. In the end, after inflicting $7 million of property damages, the Killdozer got stuck in a building’s basement, and its engine finally gave up. Unable to move, Marvin Heemeyer’s two hour rampage was over and he ended his own life with a handgun.
No one was injured in the incident, and Heemeyer was the only casualty. Some witnesses reported that Heemeyer consciously avoided injuring humans, with the intention of causing only property damage. He used the rifles installed on the Killdozer to shoot propane tanks and transformers. Other accounts tell that Heemeyer fired his rifles at state troopers and the wheel tractor-scraper driver.
Another incident resembling the Killdozer-case, involved a M60A3 Patton tank, and likely inspired GTA games to include tanks.
VR system with optical full-body tracking
Inception date: ~10.2015
Having experience with combining VR headsets with full-body tracking, I've been anticipating for several years for consumer VR systems to start incorporating such technology. Immersion is greater in VR when your avatar mimics the pose of your body, and social VR also calls for full-body tracking. Therefore it is just a matter of time when real-time mocap technology will be integrated with high-end VR systems like Oculus Rift, Vive, and PSVR. I expect this to happen by the 2nd or 3rd generation of these devices...
At the moment most full-body tracking systems require wearing a mocap suit or individual markers. Unfortunately that is too much hassle for a consumer product, even if it were just three additional Lighthouse tracking pucks. A markerless, optical full-body tracking system like Kinect eliminates the need for additional wearables. Regrettably Kinect tracking has too much latency and it is not robust enough for VR. But a system like Kinect can be improved upon, when using more than one sensor and better algorithms.
In 2015 when I read about HTC Vive and played around with its development kit, I thought about how convenient it would be to incorporate depth cameras into the Lighthouse base stations. One could create a sensor fusion algorithm that combines data from the two depth cameras and the Lighthouse sensors: the poses of the VR headset and hand-held controller could be directly used to get avatar’s head and hand poses and to restrict the solution space for inferring the body pose from the depth camera data. In other words, the Lighthouse sensor data would act as a prior for the full-body tracking algorithm.
While this approach doesn’t require any additional base stations, it would necessitate connecting the base stations to the computer, or at the very least to each other. Furthermore, each depth camera would most likely require their own ASIC or FPGA to perform all of the depth image processing (Kinect v2 does some of this with GPU), and a custom chip might be needed to speed up the sensor fusion. All this adds to the cost of the base stations, which already is too high. Moreover, it would be ideal for achieving smooth tracking if the depth camera frame rate would be close to the VR headset frame rate (90-120). At the moment most depth cameras have a frame rate of 30 frames per second.
These high requirements for software, hardware, the fact that Microsoft has a huge portfolio of full-body tracking patents, and the limited profit expectations even in a good scenario have kept me from pursuing this idea further. Recent developments in full-body tracking with RGB-cameras make this idea appear much more feasible though.