ABSTRACTThe usability factor of any User Interface is judged by the amount of control that is granted to the user. The more control, the more the UI is found to be useful and hence more preferred. This is particularly true when it comes to a UI which is implemented using augmented reality. As humans use their sense of touch to help in their spatial manipulation, there is a need to integrate this into a UI too.
A technology that can help to achieve this is tactile or haptic technology. The added plus of this combination is that its use is twofold. The first is that it can form full-fledged user interface that is immersive. The second is that it can create a comprehensive educational system that is inclusive and immersive.
Visual learning is the best way for learning in humans. This is because, visuals tend be retained in the memory for longer periods of time. This phenomenon can be further increased exponentially if the object of learning can be sensed by the student in more than one way simultaneously.
A good channel for this is through by using augmented reality along with tactile interfaces by which the students can see and touch the things they are learning about. The purpose of this dissertation is to analyse this new technology and to find out whether a UI made by combining augmented reality and tactile controls is feasible and whether it can be applied into education. AIMTo deduce whether augmented reality when coupled with tactile interfaces can yield a more immersive, inclusive and comprehensive education system. INTRODUCTIONWHAT IS A USER INTEFACE?User Interface or just interface is a term used to define the intermediary platform between a human and a machine. All and any such intermediary layer is called an interface. Interfaces include but are not limited to switches, toggles, scales, and metres. Each a very type of machinery or technology requires an interface so that the user or the human operator can interact and operate it.
Technology can be found in every corner of the world and in every aspect of life, yet, we still persist on becoming more immersed in this technological world. One of the leading ways for this is through an augmented reality, where the user is allowed to interact with virtual objects and perform tasks as though they would in the real world. The cutting edge of this is haptic technology. This is a technology where the user can actually feel the objects that they interact within the augmented reality. While using a tactile interface, the user can feel each and every control; this, in turn, lends itself to more control for the user while they manipulate said controls.Before we get into the details of both these technologies, let us see what each of them is. AUGMENTED REALITYAugmented reality is an indirect or direct view of a physical real-world environment in which certain elements are augmented by computer-generated or extracted real-world sensory input.
This input can be sound, video, graphics or GPS data. This technology though very new is getting more and more popularity due to the new and innovative apps and games which use it. These games and apps as mentioned earlier use a real-world environment with a digital computer-generated image or graphic superimposed on it.
1 Augmented reality apps can be classified into two main types- Apps that have tracking based on markers and Apps that have tracking based on the environment or elements of it or are marker less.Markers are in the case of AR refers to a fiducial marker. A fiducial marker is an object which can be placed in a scene so as to provide a fixed point of position or scale. In AR these markers form an interface between the real world and the augmented reality content; this is where the 3D objects, videos or images are placed.The detection of a marker by the device being used happens quickly but it happens in the following way. The calculation of the pose of the camera is achieved through a two-step process.
The first step is the detection and confirmation of the marker. The resulting information is then used to calculate the pose of the camera. The detection of the marker happens in a number of stages. The first of the stages is the pre-processing of the camera feed. The important part of this step is the conversion of the image into greyscale. This is because there is an increase in the processing speed of greyscale images when compared to that of the coloured images. There is also the fact that the information from the greyscale image is just as good when it comes to being used for marker tracking.First the camera image data is taken and then it is used for calculating the feature points from all potential markers.
These potential markers are then compared to the pre-processed data and then checked for any matches to the markers the device is looking for. Once a marker is found, the coordinates of its feature points are used to mathematically calculate the pose of the camera. 2The following is a commonly used marker named Hiro. It can be used in augmented reality apps or websites to Here is an example for marker based augmented reality app. 3 The above image shows an app named Ink Hunter which works by using augmented reality. The app shows the user how a new tattoo will look on them before they actually ink it in. 4 Above is the simple AR marker which is used to denote the point in the environment where the computer generated image is superimposed. 5The above image shows the earlier mentioned superimposition of the computer generated image of a tattoo.
As it is seen in the above images, augmented reality is of various types. It can be a simple filter applied on a photo to a complete 3D object superimposed on a real environment. As explained earlier, AR works by superimposing digital onto a real environment.
This very useful when it comes to education as it can help the students learn through a visual medium. Even though visuals can be achieved through a simple video or animation, the effect of the lesson taught using AR and the memory imprint of that lesson on the students last longer.Tactile InterfacesThe tactile interface is an interface which works by incorporating tactile or haptic technology. This is the technology by which users will able to actually feel the virtual environment they are interacting with and its elements.This technology though quite new can prove to be very useful when implemented in virtual and augmented reality interfaces.This is due to a number of reasons but the main reasons are threefold.
1) They can grant more control to the user.2) It can form a UI which is inclusive towards people who are visually impaired.3) It can be used to convey information to the user using a new channel, touch.But this technology requires special equipment to be used. These are called tactile transducers. They are small electrodes or electrical/pneumatic devices that when activated send an electric shock, vibration or pressure onto the body of the user. These receptors will then be built into wearable technology like gloves which can then be used to control and interact with the interface.
The tactile signal can be fine tuned in such a ways as to convey anything from a tap on the shoulder to a blunt force or a resistant force, like the one which is felt while holding an object. This can be done by creating variations in the position, amplitude, frequency, waveform type, and the tactor’s duty cycle. Similar effects can also be achieved by using different types of tactors or different types of stimulus like pressure or resistance to motion.
But the factor that plays a key role in the conveying of information is not the type of stimulus or its different varying forms; it is, in fact, the mapping of these stimuli. The mapping is very important because it will be too much information at the same time if the tactors are too close together or produce stimuli that are too similar to each other. Similarly, if they are too far apart they can never convey a recognizable tactile feedback. 6The above image shows the tactile transducers or tactors which convey the information from the interface to the user through their sense of touch. This is just one method of introducing tactile feedback to the user, there are many others.
The others methods of feedback can be through wearable technology with vibrating transducers, or inflating bags as mentioned earlier or it can be through free air transmission or through proximity vibrations.Regardless of the medium through which the tactile feedback is conveyed to the user, the main aim of the technology is to convey information to the user using the feedback. There are three important aspects to tactile technology which are a necessity to achieve proper tactile feedback. The three aspects are tactile presentation, tactile stimulation, and Tactile Language.Tactile Presentation: -This deals with how the information is presented to the user. It can be split into two types- passive touch and active touch.
Passive touch is how the information is presented passively to the user’s skin. This is the most commonly used type of tactile presentation. Active touch is how precise information like the shape and texture of an object is conveyed to the user.
This used when such precise information has to be conveyed to the user without the use of a visual medium. It is also important if the user’s visual focus has to remain on a particular point while simultaneously receiving information about the second object. It is also used as a subconscious directive about a certain function, for example, if a certain flat surface needs to be tilted up or down the control surface reflects the shape of the object so that the user can use the shape and an instruction or directive to coordinate and control its movements with the control surface.
Tactile Stimulation: -This deals with what type of stimulation is given to the user. It describes different tactile channels like electrical, electromechanical and pneumatic.Electrical stimulation means applying a small electric shock below 0.000001 Joules.
This is the threshold for electrotactile stimulation, but the issue with this is that the Pin threshold is very close to the electrotactile threshold and so it is often very uncomfortable for the user.Electro-mechanical means producing a sense of touch using vibrations or other physical stimulants on the users’ body. They are very useful for producing two-point stimulation without pain or any discomfort.Pneumatic devices use bladders or pockets that inflate or deflate at a rapid pace to produce a sensation of touch.Tactile Language- It has been estimated that even experienced Braille users can perceive 25 to 50 bits/sec and so a visual medium is preferred for conveying information to the user.
But tactile channels can be used when visual channels are overloaded or to supplement other channels. Due to this fact a special language has to be designed so that information transmission is possible. This language is what determines which part of the user’s body to stimulate and the type, number, range, intensity and the spatial distribution of the stimuli.
This language creates certain rules and syntaxes which assign certain feedbacks to certain stimulants. As mentioned earlier all these aspects have to be used properly to achieve tactile feedback, but the bottom line is still that the feedback has to distinguishable and understandable to the user. Tactile technology is both new and old as it is only now entering the digital space but it started from Braille and other older modes of tactile information exchange. Some of the earliest works in Haptic research date back to the 1800s.
In that that time period researches worked with the idea of sensory substitution where one sense is used to receive information which under normal circum stances would be received by another. This era pioneered a system named Elecktoftalm where blind people could differentiate between light and dark by using a photosensitive element called Selenium.The field of haptic perception or tactile feedback gained more innovations in the 1960s as well. It was in this time period that the Elecktoftalm was modified to provide tactile stimuli for the user.
The 1960s also saw the creation of the Tactile Vision substitute system which would send impulses to user using tactors attached to their back when the sensor detected changes in the brightness of its view field. There were also many other innovations such as the optical image to tactile conversion unit and the tactile television system.As time moved forward there were many other new inventions such as the tactile reading aid for blind users which enabled them to read and consume content such as daily news and serialized fiction. Tactile technology is currently being developed for use in gaming and other related sectors in order to achieve a better immersive gaming experience for the user.RESEARCH METHODOLOGYCASE STUDIESThe case study will be the research methodology used to achieve the aim of this paper.
The items reviewed will be the latest articles and features related to the topic.1. Gloveone- https://www.roadtovr.
com/2-vr-gloves-promising-haptic-feedback-2-different-approaches/ 7The above is a link for a video which discusses the different aspects and possibilities of the new haptic glove technology developed by NeuroDigital Technologies. NeuroDigital Technologies is a Spain based start up technology firm, which has developed this revolutionary new tactile device called Gloveone. It is a wearable technology in the form of a glove. It uses 10 transducers or tiny actuators which are spread over the glove.
The actuators vibrate independently at different frequencies. Their intensity of vibration can also change and hence they can convey the sensations of touch accurately. The feedback from these transducers are received and transmitted through a USB connection or by using a Bluetooth module powered by a 800mAh battery on board the glove, which can provide 4 hours of battery life. The glove cannot actually create actual touch it can only simulate it by using the approximate vibrations that can act as a placeholder for the real touch interactions. The glove also has a 9-axis IMU hidden behind the logo, which is used in tandem with other commercial sensors like Leap Motion and Intel RealSense for hand tracking, which in turn provides stable interaction for the user.
2. Hands Omni- https://www.youtube.com/watch?v=QaiGAI7ZeEY The Students of Rice University well sponsored by the technology firm Virtuix to develop a haptic glove prototype. They were the ones who came up with the Hands Omni device.The Hands Omni glove uses a different method to convey the touch sensation to the user. Instead using vibrations or tactile transducers, they use mini-bladders which inflate to convey pressure which in turn is detected by user as an approximate sensation of touch.
This method is quite interesting as it provides an actual physical sense of pressure to the user that the mini actuators cannot. But in terms of the effectiveness of the Omni glove the verdict is still out as the creators’ state that the Hands Omni glove is only at a development and miniaturization level similar to that of the Google Cardboard. The touch sensation conveyed by the Hands Omni Glove by using inflating bladders is much more effective the normal mini-rumble packs as it can actually convey the sensation of force and pressure that the user feels when holding a real object. Also, as it doesn’t opt for a vibration feedback the user finds it much easier to actually feel the virtual object rather than just receiving an approximate touch sensation.The technology lags behind it contemporaries only because it is currently under active development and also due to the fact that the technology itself at present is bulky and yet to be miniaturized. 3.
Senso- https://www.youtube.com/watch?v=ycg7dG45Z4A 8Senso is a neww Haptic glove technology which offers per-finder haptic feedback and alsso the baility to simulat temperatures. The glove has a 7 IMU system and 9 axistantial sensors to Facilitate 6FOD positional tracking with no ther standalone solutions.
9Each finger of the glove has a vibrational motor packed into to convey sensation better. But much like it’s contemporaries the Senso glove too suffers form tracking usses like immersion breaking drift and innaccuracies. The gloves are said to be bale to survive for upto 10 hours via the built in battery. LITERATURE REVIEW1. Tactile User Interfaces techniques and Systems to NASA Ames Research Centre by Lily Spirkovska.The main literature collected for research on this topic is the thesis paper submitted under the name –Tactile User Interfaces techniques and Systems to NASA Ames Research Centre by Lily Spirkovska.
The paper mainly discusses the origin, history and the ongoing development of tactile technology. It also discusses the different ways in which the technology can be used and how it interacts with the users.It also describes the three main aspects of the technology which are Tactile Language, T tactile simulation, and Tactile Presentation. Even though these topics go deep into how the technology and its signals will be interpreted by the user and tends to leave behind the actual technological part and how it can work with other interfaces, it also gives a lot of insight into how the technology works, which in turn can be used to find out how and what parts of can integrate well with the AR technology.2. AIREAL: Interactive Tactile Experiences in Free Air by Rajinder Sodhi, Ivan Poupyrev, Matthew Glisson, Ali Israr.
The second literature collected for research is the thesis paper submitted under the name- AIREAL: Interactive Tactile Experiences in Free Air to Disney Research in Pittsburgh by Rajinder Sodhi, Ivan Poupyrev, Matthew Glisson, and Ali Israr. The thesis paper describes a new technology named AIREAL. It is a new haptic technology which can create and deliver effective haptic feedback without using a delivery device. This means that the user can feel the tactile sensations without wearing a device like a glove or any other wearable technology. The paper discusses the different aspects of the technology and how the user will be able to play games or use a device interface without being terrestrially bound to the device.
1 https://boygeniusreport.files.wordpress.com/2017/02/augmented-reality-iphone.jpg?quality=98&strip=all&w=782- Accessed on -13/9/20172 https://jeromeetienne.github.io/AR.js/data/images/HIRO.jpg3 http://www.androidauthority.com/best-augmented-reality-apps-and-ar-apps-for-android-584616/- Accessed on -13/9/20174 http://www.androidauthority.com/best-augmented-reality-apps-and-ar-apps-for-android-584616/- Accessed on -13/9/20175 http://www.androidauthority.com/best-augmented-reality-apps-and-ar-apps-for-android-584616- Accessed on -13/9/20176 https://www.roadtovr.com/2-vr-gloves-promising-haptic-feedback-2-different-approaches/ -Accessed on-2/9/20177 https://www.neurodigital.es/store/68-home_default/gloveone.jpg8 https://roadtovrlive-5ea0.kxcdn.com/wp-content/uploads/2016/12/Senso_2.jpg9 https://roadtovrlive-5ea0.kxcdn.com/wp-content/uploads/2016/12/Senso_inside.png