Introduction
Problem Definition and Motivation
Main Problem:
Nowadays there are many VR applications with complex menus and capabilities,
giving hard time for users to adjust and finally give up on using the app.
Goal:
Our project's goal is to overcome this challenge and present a new way of interacting with an application via recognition of motion and shape instead of menus and buttons.
Our solution will allow to link the controller motion in 3d space to a certain behaviour in the application.
In our case we will implement the idea as a game in which the user plays a wizard that can draw his spells as shapes and for each one of them a predefined spell will be activated.
Solution
Platform :
•Hardware platform: HTC vive
•Software platform: Unity 3D,VS 2017,Python,Mnist/Tensorflow
​
Python:
We have created a python application that serves as a server that receives and analyses a shape/picture and send an answer regarding the identity of the shape to the client.
We have used Mnist/ TensorFlow algorithm for shape recognition. We have trained the algorithm to recognize 3 different shapes and build an interface for image input and output via TCP sockets.
Unity:
On unity side we have separated the VR world into 2 different space:
1.The user space – the space which can be seen by the user:
a.The environment
b.The enemies
c.The drawn shapes
d.The magic spells
2.The image processing space – the space which includes the setup for processing the shapes drawn by the user:
a.A blank canvas for plain background
b.Line drawer component for redrawing the shape from user space
c.A camera to capture the redrawn shape
Demo
Improvement ideas
​
The following ideas might be implemented as part of future projects:
​
• Expand the shape recognition database to support more shapes.
• Implement an interface to receive shapes images as user input and expand the database according to his unique drawings.
​
• Expand the feature to support 3D images/models.