Tumbnail3.png

Whack-A-Holomole

 

OVERVIEW

Whack-A-Holomole is a Microsoft Hololens gaming application, which aims to change turning on and off lights, a boring task, into a fun and intriguing experience. Players will whack moles, which will constantly turn on the lights and change the color of the bulbs, until they finally manage to turn off the unnecessary lights.

- Presented demo to TechSAge State of the Science Conference 2017

1.MOTIVATION

Augmented reality provides a unique experience because not only does it show immersive 3D graphics, but also the 3D graphic images are placed in the real world. I think the experiences AR technology creates can be enhanced if the 3D graphic images interact with real world objects, so I wanted to connect Microsoft Hololens with smart home devices, such as smart plugs, smart bulbs, and WiFi thermostats.

I had two scenarios for this project. The first idea is an educational game that will help people understand how IoT devices work. The second is to help people engage with smart devices and energy-saving activities.

2.BACKGROUND

2.1. AUGMENTED REALITY AND EDUCATION

This project started with the idea that AR could be used as a learning tool in education and everyday life. I hoped that the immersive feeling of the AR environment would raise educational efficacy and interest and intrigue children who otherwise become bored easily while studying.I researched educational AR applications that are currently being developed. Even though the subject and techniques applied were different, the apps enable users to interact with virtual screens or objects in the real world, which make the experience very special for students.

2.2. IOT (INTERNET OF THINGS)

IoT technology is more advanced than Internet or mobile Internet technology based on existing wired communication, and devices connected to the Internet communicate with each other without any human intervention. In addition to home appliances and electronic devices, information can be shared by networking objects in various fields such as the healthcare, automotive, and smart home fields.

The most widely used environment for IoT technology is the residential space, which is called a ‘Smart home’. This means home appliances, lighting, energy, and security functions as well as smart cars and smart education.

3.DEVELOPING

3.1. ENVIRONMENT SETTING - CONNECTION TEST

I wanted to incorporate IoT technology with AR to improve the home education environment. So, the first thing I needed to do was make a connection between the Hololens and smart devices. Firstly, I made a DIY-style smart plug using Arduino and Relay. I was also able to connect Philips Hue smart bulbs to the Hololens. Lastly, I connected the Nest thermostat to the Hololens. I used Unity3D (C#) and RESTful API to control the Philips Hue smart bulbs and the Nest thermostat.

 [Hololens + Smart plug using Arduino: Smart plug using Arduino used for this test]

[Hololens + Smart plug using Arduino: Smart plug using Arduino used for this test]

 [Hololens + Nest Thermo: Nest thermostat that is used for this project and connected to 24V external power adaptor]

[Hololens + Nest Thermo: Nest thermostat that is used for this project and connected to 24V external power adaptor]

Finally, In this project I chose Philips Hue because it supports dimming control as a default feature as well as a color change feature, which may provide various effects for the future prototype. To create a gaming environment, I replaced the lightbulb in the room with a Philips Hue lightbulb which can communicate with the Microsoft Hololens. When a participant wearing a Hololens gazes and air-taps, a lighted bulb turns off in response to the participant’s behavior.

3.2. POSSIBLE GAMING SCENARIOS

The motivation for this project, as part of the whole home gaming project, was to make an entertaining experience, which will enable people to move from room to room or space to space in their home. The reason I chose the Hololens headset as a base technology for this project is that I believe the Hololens headset can create an immersive and explorable experience which makes the ordinary home environment into an amusing space.

While testing the IoT devices, I had another idea to motivate people to engage with smart devices and energy saving activities. Since Philips Hue can provide a colorful, party-like environment, I thought it might be a delightful experience if they can play with holograms that are synchronized with smart bulbs.

3.2.1. SCENARIO 1: LET PEOPLE KNOW HOW SMART DEVICES WORK

For this scenario, holograms can be used to visualize the invisible relationships between smart home devices. It can be a powerful educational application because the Hololens headset is very good at showing something otherwise unseen, such as pipes behind walls.

When the user starts the application, it shows the location of smart hubs, such as Amazon Echo or Google Home, and they can select one of the devices using gaze and air-tap gestures. This application also teaches users about basic troubleshooting processes. The most common problem with smart devices is “connection lost”. Holograms can be used to visualize where the lost connection occurred and show users the steps they have to take to fix it.

Des06.png

3.2.2. SCENARIO 2: WHACK-A-MOLE + SMART BULBS

The purpose of this scenario is provide an entertaining experience by playing an augmented reality game which is connected to smart bulbs. It also aims to provide a message about energy saving. The reason I choose ‘Whack-A-Mole’ is to adapt a media convention that encourages people to naturally engage in the AR experience it provides. When a participant plays the game, a mole that moves up and down in the hole in front of their eyes appears, and the participant knows how to engage in the game without any explanation. The game begins with the mole appearing under the lit light, and the participant catches the mole with gaze and air-tap. If the participant catches the mole, the light goes out and they win the game.

Des07.png

4. PROTOTYPE

I used a Hololens which is connected to two Philips Hue smart bulbs via REST API to create a working prototype. The hologram is placed under the lights and consists of 5 holes where the moles will appear. When a mole appears, a light is lit, and when players catch the mole using a gaze and air-tap gesture, the light is turned off.

5. EVALUATION

To iterate this prototype, I conducted a quick-and-dirty play testing with 5 playtesters using a sampling of Georgia Tech graduate students. Testers were asked to play the prototype for 30 minutes while I observed their responses. They were asked to use a think-aloud protocol during the test. I allowed them to play it with minimal explanation because I did not want to bias them by telling them what the point of this game is and how it works. I also tried to observe their mistakes to find broader areas of improvement. I also had a post-test interview session with testers to understand their experience and opinions.

 

In general, the feedback from the testers was positive. They thought the connection between the hologram moles and colorful lights was entertaining. However, they also provided several constructive opinions regarding this project.

 

First of all, I realized that Hololens has a critical usability issue regarding calibration. I observed two out of the five testers had difficulties with seeing the holograms properly because of a combination of the Hololens’s narrow field of vision and difficulties with wearability. This was a problem I also observed with casual tests with my friends. It was a critical issue because they were not able to play the game properly with this problem. In this case, their problem can be solved when they go through a calibration session using the calibration application offered by Microsoft.

Most of my testers mentioned that the gaze and air-tap gesture may not be the best interaction modality for this game. A tester said that the gaze gesture is difficult because using the movement of your head is not intuitively connected to moving a cursor because conventionally we use our hands to move cursors. Some testers told me that the air-tap gesture was impressive because of the high rate of cognition, however, it was difficult to get accustomed to the gesture. Another player suggested an idea of using a Wii controller or a smartphone to imitate a hammer, which is usually used in the traditional whack-a-mole game.

Lastly, they pointed out a lack of general user interface elements, such as a score section, a help button and a menu button. I am currently testing different types of general user interface elements, which are optimized for the Hololens application and will update this prototype in the near future.

6. FUTURE WORK

I was able to more intimately learn a lot about the utility and functionality of the Hololens by developing this application. Even though the prototype has some areas for improvement, I believe it is a worthy project to explore.

 

I found areas for improvement during development, demo, and user testing. Firstly, I would like to add levels to this prototype to expand the users’ experience and playtime. Additional user interface elements for games, for example scores, a how-to-play screen, and ending screens, can help users become immersed in the game environment. As I learned from the user testing, it can be improved in terms of its educational aspect. The testers mentioned that the game mechanism is interesting and amusing, but it is little bit difficult to learn about energy issues. I would like to apply storytelling to the beginning of this game to offer educational value to players.