1. Built Arduino glove with Bluetooth module and vibration motor
2. Connected the glove with Android Unity application by Bluetooth communication
Team of 3 for 16 weeks
Shin-young Mo (Programmer)
Min-su Kim (Programmer)
Google Cardboard, Kinect, Unity3D (C#), Arduino
First Prize Winner in Graduation Exhibition of Computer Science
Semi-Finalist in 13th Yonsei Creative Exhibition held by College of Engineering
This project aims to improve current VR interface by designing a new input and output device. Aside from HMD (Head Mounted Display), several input devices such as keyboard and gamepad are being used. However, most of these devices are not appropriate for VR interface since they are not intuitive and require users to pay attention to the devices constantly. For instance, if a keyboard is used as an input device for a VR game, the player should remember the position of the buttons and have a restricted space to move. Thus, in order to fully utilize the space and increase the sense of reality we developed an input device with Kinect sensor so that the user can naturally send input values using hand gestures. In addition, there are few output devices that could give a feedback to the users experiencing virtual reality. To enhance VR interface with a realistic feedback, we designed an output device using Arduino and vibration motor.
There were several existing works regarding VR interface, but in majority of those works, the users could not utilize the space fully enough to move freely due to electrical cords. In contrary, we developed a system that provides cordless communication between Android and Kinect. In most cases of existing works, Oculus Rift was used so that the electrical cords were unavoidable. We designed WiFi communication between Android application and Kinect thus users can experience wireless interface using cordless HMD such as Google Cardboard.
Existing controllers used as an input/output device for VR provide haptic feedback with no variation to users, since controller vibrate as a whole. However, we planed to develop a wearable output device which can provide various haptic feedback. According to the position of vibration motor, users can experience diverse feedbacks. For example, when a user touches a certain object in VR scene with his/her two fingers, only two vibration motors attached to those fingers will vibrate. This suggests possibilities of developing more realistic and natural output devices for VR.
Kinect sensor detects the users’ movements and hand gestures. Then, it conveys the input signals to PC. In PC, Unity project analyzes the signals and sends those input values to the VR scene of the Android application via wifi connection. Since the Android application cannot receive the input values of Kinect sensor directly, network communication between PC and Android is inevitable.
In order to make the input device, we confirmed that the local host can convey PC client data to local host server successfully. Then, we installed Kinect extension plugin in Unity so that the PC client can analyze the input signals that comes from the Kinect sensor. Analyzed data is sent to Android application and is used as input values of VR scene.
We developed the wearable output device using Arduino Uno and vibration motor. Arduino gives users different haptic feedbacks depending on the specific conditions through Bluetooth communication between VR Android application and Arduino.
Testing Bluetooth communication from Android
First of all, we connected Bluetooth module to Arduino Uno and checked if PC can detect and communicate with Arduino through Bluetooth module. Since there is no library that is able to handle Bluetooth communication in Unity, we designed our own Bluetooth plugin using Android Bluetooth function with Java. Then, we built an Android application using Unity that can control the vibration motors of Arduino. We attached the Arduino on gloves and installed the vibration motors on the tip of each finger.
This is the simulation video of our project. The background of the simulation is the universe with objects that illustrate various planets. The input device of our system, Kinect sensor can detect and recognize whether a player clench his/her fist or open his/her hand. Thus, the player can touch the planets, hold them and move them. White ring-shaped indicators signify the current position of the player’s hands in the VR scene. When the player drags or drops a planet, vibration motors are activated. Different length of vibration is activated depending on the size of the planet. If the sun is dragged, the vibration length will be longer than that of Earth. Many different variations can be added to the system. For example, vibration motors can be activated not simultaneously but sequentially with a particular order, since the vibration motors are attached to each fingertip. Also, depending on the conditions, vibration motors can be partially activated.
We conducted a user testing to evaluate our system in terms of interactivity and sense of reality. In this user testing, interactivity refers to the degree to which users agree to the conditions that give them a haptic feedback while experiencing the new system. Sense of reality signifies how much users feel real compared to existing system.
We designed an experiment in which participants were asked to play the VR game we used in the simulation twice – with and without our input and output devices. We recruited eight undergraduate students and divided them into two groups. Those who are in the first group played the game with our system first, then played without it. The second group are required to play the game without the system first, and then with the system. After the user testing, we conducted the individual survey. The questions were made up to see the satisfaction difference in terms of interactivity and sense of reality.
According to the survey conducted, average point of interactivity was 3.8 out of 5 and that of the sense of reality was 3.3 point. This indicates that majority of the participants were positive about our system. Limitation that some of the participants suggest was that the response time was quite slow due to wifi connection.