3D+Stereo Augmented Reality for Landing Systems Feasibility and Experiment Design

This paper considers the possibility of creating an aircraft landing system (LS) with the augmented reality (AR) 3D+stereo interface (patent pending). Such a LS system can be used as a supplementary navigation resource and has the following advantages in comparison with the existing solutions: improved spatial navigation, shirt pocket size, autonomous power supply, minimal training, and low cost. The design of experiments to prove the effectiveness of an AR 3D+stereo interface is described.


Introduction
Possibilities provided by the AR technology correspond well with the specifics of the LS used for landing approaches in limited visibility conditions. This statement is confirmed by the following examples dated 2010: 1. An Enhanced Vision System (EVS, developed by Rockwell Collins) [5] has been certified by US Federal Aviation Administration for Boeing aircraft. This system includes the head-up display ( fig. 1) which is per se an "AR" device with 2D virtual objects (speed vector symbol etc.).

Problem
AR in LS requires the use of a special display device which combines virtual objects with the real world. In the systems mentioned above (as well as in all others) such displays are cumbersome, very high cost solutions and in addition these devices do not provide a pathway in the system's development in the direction which is obvious now: from 2D and 3D to 3D+stereo.

Solution
At least two companies -Laster Technologies [3] and Lumus [4] have announced last year the AR display devices looking as common glasses with transparent lenses containing special optic elements which show the user a stereo image of virtual objects. Such an approach transforms AR stereo displays into small and comparatively cheap glasses ( fig. 3 and 4). Both companies promise to make their products commercially available in 2011.  The combination of such AR glasses with a mobile device containing GPS, accelerometer, digital compass, gyroscope and high performance processor (e.g. iPhone4) provides everything necessary (6DOF tracking, processing tool and stereo AR display) for the creation of the pocket size (literally -fitting into the shirt pocket), autonomous and low cost LS which could be used as a back-up or supplementary on-board navigation system. The necessity of such a system was proved by the crash while landing of the plane in Moscow Domodedovo airport in Desember 2010. All the plane's engines failed, as a result the on-board electrical system also failed, and this by-turn caused the failure of the on-board navigation system. Because of very low clouds and poor visibility the pilots carried out a "blind" landing using only the air traffic controller instructions resulting in catastrophe. Also the value and the relevance of such a pocket LS is obvious for light aircrafts.
Besides the above-stated, the suggested LS allows implementation of a 3D+stereo visual interface which is most natural for a human being. This interface not only improves the quality of the pilots spatial navigation, but minimizes training required for using this tool. A pilot will be able to see the approach descending path as a series of markers that can be represented by 3D frames in the stereo mode ( fig. 5). The flight inside the virtual tunnel marked by the 3D frame increases the probability of an error-free landing in poor or even zero visibility. A kind of such a tunnel is already realized in systems known as synthetic vision systems (SVS), but these systems provide non-stereo 3D virtual reality without the visual contact with the real world [6,7]. Smart-View SVS developed by Honeywell [2] has AR display combining infrared camera data with virtual reality, but it also supports only non-stereo 3D and 2D (as well as Rockwell Collins's EVS which allows showing monochrome 3D terrain models or infrared camera picture).
The problems of GPS accuracy and accessibility are out of the scope of this paper. Let us point out briefly that the current state of GPS (and differential GPS) and similar systems (GLONASS) and especially the perspective of these systems integration in the nearest future make the satellite positioning acceptable for obtaining linear coordinates for a LS of supplementary type. GPS is used in Honeywell's SmartView for 3D terrain model positioning. By 2009 US Federal Aviation Administration registered more GPS landings than landings with "traditional" instrument LS. iPhone4's built-in gyroscope and accelerometer solve the problem of angular coordinates.

Experiment Design
Despite the intuitive evidence for the high effectiveness of 3D+stereo interface (in comparison with 2D and 3D) for LS, a study proving its advantages is needed (or the absence of such an advantage). In order to do it in the beginning of 2012 an experiment series will be carried out in the Moscow State Technical University of Civil Aviation. Three groups of subjects (each group consists of 12 students, groups are homogeneous by gender, age and education level) are to take part in the experiments. Pilots intentionally are not included in the subject groups, since their professional experience of flying the plane using the present instrument technology would be a distorting factor for the comparative experiments.
Three computer flight simulator programs have been developed using the Vizard VR Toolkit [8] to simulate the flight along the same landing path: the flight using AR LS with 2D interface (like shown on fig. 1); the flight using AR LS with 3D interface (as shown on fig. 2); the flight using Runway Landing path markers AR LS with 3D+stereo interface (as shown on fig. 5). The virtual plane is controlled by means of a joystick. 2D and 3D simulators work with a common display, 3D+stereo simulator works with a stereo display (ViewSonic vx2268vm 3D Ready and NVIDIA 3D Vision Glasses). The 1st group of subjects "lands" the virtual plane using the 2D interface, the 2nd group of subjects uses the 3D interface, the 3rd group of subjects uses the 3D+stereo interface, thus the independent variable is the type of the visual interface. The dependent variables are the mean-square deviation of the virtual plane trajectory from the ideal landing path and the mean time spent by a subject to correct deviations from the ideal landing path arising due to the modelled external actions. The simulator structure is shown in fig. 6. The experiment results processing scheme is shown on fig. 7. The statistically significant difference between the experimental results is determined by the analysis of variance (ANOVA).
All subjects have small instruction and learning session before the trial.
Processing procedure for trial results is important part of the experiment design. Mean-square deviations are general, static estimates for the random process of the divergence correction. In order to have the estimate of this process dynamics, it is presented as a Markov chain [1] which includes 3 states ( fig. 8): S A -the state when the divergence between the virtual plane trajectory and the ideal landing path ascents; S D -the state when the divergence between the virtual plane trajectory and the ideal landing path descents; S C -the state when the divergence between the virtual plane trajectory and the ideal landing path does not exceeds some small threshold T.
Let the divergence at the moment i be R i . One can describe changing of the states of the correction process by means of a time series consisting of R i . Then it is possible to interpret the suggested chain as an approximation of a random process with unlimited countable number of states by the first differences A random process is called a Markov process if its future evolution depends only on the current state and does not depend on the trajectory, which has led the process to the current state. The behavior of a subject while correcting the flight trajectory does not depend on his/her previous correction actions so it is reasonable to consider the correction process as a Markov one.
During the simulation the numbers of transitions between states S A , S D and S C are registered as the events of changing the sign of difference R i -R i-1 . After the generation of the statistic estimates for the elements of matrix {π ij } the equation system (1) is solved to find the final probabilities P A , P D and P C .
These probabilities are informative parameters of the correction process dynamics. The correlation of P A , P D and P C in conjunction with the mean-square deviation provides a rich spectrum of useful interpretations of the experiments results. For example big shares of P A , P D even with small mean-square deviation definitely mean the impulsive control actions, and that by-turn means the discomfort of the visual interface for the subject.
Even more analysis possibilities may be obtained if the divergence correction is considered as a semi-Markov process with embedded Markov chain. Semi-Markov process differs from Markov process in the sense that the future evolution of the first one depends not only on the current state, but also on the time of being in this state. These times affect the final probabilities and this may improve "the analytical resolution".

Conclusions
In case of confirmation of the 3D+stereo interface effectiveness, the perspective is opened for AR LS with the following advantages:  Improved spatial navigation.  Pocket size.  Autonomous power supply.  Minimal training.  Low cost. It could be a market breakthrough for the field of industrial AR applications. So far AR technology products are being used extensively in the entertainment sector and for simple navigation using smartphone apps. Pocket size AR LS with a 3D+stereo interface will be inevitably popular among small/light aircraft pilots whose number worldwide (1) makes it reasonable to discuss the considerable market of fully utilized AR solutions