How to Apply Real Bat and BallintoVirtual Baseball Contents

It is very challenging issue to utilize human-being’s behaviours or reactions as a way of interacting to virtual world. Indoor baseball simulation system is one of good candidates to combine real equipment and actions with virtual contents. As one of experimental approach to break a barrier between virtual game space and human-being’s real behaviour, we mapped out an experiment on seamless gaming between 3D virtual game space and human-being’s real behavioural space, where real balls and bats are augmented with online baseball contents through computer vision and sensor technologies. The Simulation Engine, the key part of our proposal, consists of 6 components: Pitch Recognizer, Toss Controller, Swing Detector, Vehicle Recognizer, Vehicle Tracker and Trajectory Calculator. This approach, utilizing human-being's behaviours as input event, will open the new field of digital interfaces.


Introduction
We are embarking on the exploration of developing interfaces that can work with missing intelligence and aesthetics [1,2,3]. To do this, interfaces must understand our metaphors, must solicit information on its own, must acquire experiences, must imp rove over t ime, and must be intelligent in context.
For a context-sensitive indoor baseball game, hu manbeing's interaction with the simu lation system must be augmented, making the computer as seamlessly as possible, by exploring the use of a real ball for interaction with the virtual applicat ion on a screen. Therefore, they may augment actions that users are capable to perform in the real world, both in quantity of tasks performed and their quality [4] Especially indoor baseball simulat ion system needs the use of real ball and bat fo r interact ion with its v irtual application to reflect on user's natural behaviour aspect as well as to make a user immerse into the game. As one of experimental approach to break a barrier between virtual game space and human-being's real behaviour, we mapped out an experiment on seamless gaming between 3D virtual game space and human-being's real behav ioural space, where real balls and b ats are co mb ined with on line baseball contents through co mputer v is ion and sensor technologies.
The remainder o f this paper is organized as fo llo ws. Section 2 co mpares with existing related wo rk. Section 3 introduces our ideas about augmented indoor baseball simu lation game interface including conceptual system structures as well as synchronization mechanis m between a virtual p itcher's pitching ball and a tossing machine's one. Finally, we conclude with a note on the current status of our projects and future works.

Related Work
A variety of research efforts have recently explo red computationally aug mented interfaces that emphasize human interaction using the human being's behaviours and senses.

Camera-based Approach
The MagicMouse allows the user to operate within both 2D and 3D environ ments by simp ly moving and rotating their fist. Position and rotation around the X, Y and Z-a xes are supported, allowing fu ll six degree of freedo m input. This is achieved by having the user wear a glove, to which is attached a square marker. Translation and rotation of the hand is tracked by a camera attached to the computer, using the ARToolKit software library [5]. Wagner and Schmalstieg use a PDA camera to recover the position of specific markers positioned in the real environ ment [6]. Rohs uses visual codes for several interaction tasks with camera-equipped cell phones [7]. His IPARLA Project designed a new 3-DOF interface adapted to the characteristics of handheld computers. This interface tracks the movement of a target that the user holds behind the screen by analysing the video stream of the handheld computer camera. The position of the target is directly inferred fro m the color codes that are printed on it using an efficient algorithm [8].

Sensor-based Approach
Affective Co mputing Group at MIT is making an effort for sensing, recognizing, understanding, and synthesizing the human behaviour patterns. This group discussed the use of biometric sensors with wearable co mputers. Such sensors allo w for new interactions between the wearable and the wearer, which they based upon affect detection, prediction, and synthesis [9].

Object Tracking
To recognize and track an object, many techniques have been devised including edge-based, optical flow-based, texture-based and hybrid methods. [9] used both edges and optical flow without the need of a known mot ion model, which is the case of most AR applications. Texture based feature extraction and optical flow tracking were also joined together in a mu ltithreaded manner in [10].

Sports Simul ati on
Visual Baseball takes 2000 p ictures each and every second, and determines the exact trajectory, d irection and speed of an object like our approach, however, it supports only pitcher mode not batter mode [11].
Our approach is the first trial in the world for batting simu lation comb ining camera and sensor technology, and synchronized with tossing machine, virtual content and batter's swing. As shown in Figure 1, our proposed system has following specification and equipment.

Simulator System Flow
When a batter enters into batter box the pitcher on v irtual contents pitches a ball. After the Simu lation Engine catches the ball's attributes it co mmands 'toss' to Toss Machine. As soon as the Optical Sensor detects whether the batter's swing or not, High Speed Camera takes over 2000 p ictures each and every second. After the Simulat ion Engine analyses and calculates it sends the ball's trajectory and speed to Virtual Baseball Contents. The 3D Contents represent the hit ball and reaction of defenders and spectators with appropriate sounds. This flow is shown in Figure 2.

Simulation Engine
Our Simulat ion Engine consists of 6 co mponents as shown in Figure 3; Pitch Recognizer, Toss Controller, Swing Detector, Vehicle Recognizer, Vehicle Tracker and Trajectory Calcu lator.
1. Pitch Recognizer: receives type and speed of a ball fro m virtual p itcher.
2. Toss Controller: sends the type and speed of a ball to portable toss machine thru infrared co mmunicat ion.
3. Swing Detector: detects batter's swing thru infrared sensor on an end of home plate.
4. Veh icle Recognizer: analyses the scene from high-speed camera and find hit ball.
5. Vehicle Tracker: tracks the movement of the hit ball and sends the position by predefined coordinate to Trajectory Calculator. We use our Color Blob Matching algorith m [12] to track the vehicle.
6. Trajectory Calculator: solves trajectory and speed of the hit ball by our algorithm, and sends them to virtual contents.

Trajectory Calcul ation
Basically, there are three co mponents of aerodynamic force acting on the baseball in Figure 4: the drag fo rce, F d ; the lift force, F l ; and the side force, F S , act ing lateral to the ball's flight path. The largest aerodynamic force acting on the ball is the drag component. The lift and side forces are caused by asymmet ries in the stitch patterns, and by the so-called Magnus force, F M , resulting fro m ball spin. Co mputations of the baseball's trajectory, including wind effects, require that the wind velocity field generated by Virtual Contents randomly. The wind velocity field (V w =V wx +V wy +V wz ) is determined by a combination of methods: hot wire anemo meter measurements in a scale model of the stadium in a wind tunnel and Computational Flu id Dyna mics (CFD) modelling.

Tossing Machine Synchronizati on
Our tossing machine has following specification and function.
•Light weight equipment wh ich takes about 20 balls; •Portable machine wh ich user can put it appropriate place; •Wireless control using infrared co mmun ications; •Synchronized with speed, direction and type of virtually pitched ball.

Reaction of Virtual Contents
To redouble feeling response of users our virtual baseball contents react to a batter's swing and hit on a ball as follows and shown in Figure 5.
•3D real-t ime realization of batter's swing and hit; •A realistic reaction representation of a p itcher, catcher, judge, defenders and spectators; •An accurate realization of h it ball's trajectory and defender's movement; •An appropriate sound effect tightly related to batter's swing and hit.

Conclusions
For breaking a barrier between virtual game space and human-being's real behaviour,we attempted an experiment on camera and sensor-based user interface for indoor baseball simu lation wh ich interacts to virtual baseball contents.We mapped out an experiment on seamless gaming between 3D v irtual gamespaces and human-being's real behavioural space, where real balls and bats are augmented with online baseball contents through computer vision and sens or tech nol ogi es. Ou r p ro pos ed Si m ulation Engineconsists of 6 co mponents: Pitch Recognizer, Toss Controller, Swing Detector, Vehicle Recognizer, Vehicle Tracker and Trajectory Calculator. Our approach is the first trial in the world for batting simu lation co mb ining camera and sensor technology, and synchronized with tossing machine, virtual content and batter's swing. These attempting will increase friendliness and immersion to a game.
We are still wo rking on following areas.
•Motion or gesture interface as new game interface •Social games thru n-screen •Interaction mechanis m to make real and virtual world seamless