Video Tracking

2

APPLICATIONS

2.1 INTRODUCTION

Tracking objects of interest in video is at the foundation of many applications, ranging from video production to remote surveillance, and from robotics to interactive immersive games. Video trackers are used to improve our understanding of large video datasets from medical and security applications; to increase productivity by reducing the amount of manual labour that is necessary to complete a task and to enable natural interaction with machines.

In this chapter we offer an overview of current and upcoming applications that use video tracking. Although the boundaries between these applications are somehow blurred, they can be grouped in six main areas:

• Media production and augmented reality
• Medical applications and biological research
• Surveillance and business intelligence
• Robotics and unmanned vehicles
• Tele-collaboration and interactive gaming
• Art installations and performances.

Specific examples of these applications will be covered in the following sections.

2.2 MEDIA PRODUCTION AND AUGMENTED REALITY

Video tracking is an important element in post-production and motion capture for the movie and broadcast industries. Match moving is the augmentation of original shots with additional computer graphics elements and special effects, which are rendered in the movie. In order to consistently add these new elements to subsequent frames, the rendering procedure requires the knowledge of 3D information on the scene. This information can be estimated by a camera tracker, which computes over time the camera position, orientation and focal length. The 3D estimate is derived from the analysis of a large set of 2D trajectories of salient image features that the video tracking algorithm identifies in the frames [1,2]. An example of tracking patches and points is shown in Figure 2.1 , where low-level 2D trajectories are used to estimate higher-level 3D information. Figure 2.2 shows two match-moving examples where smoke special effects and additional objects (a boat and a building) are added to the real original scenes. A related application is virtual product placement that includes a specific product to be advertised in a video or wraps a logo or a specific texture around an existing real object captured in the scene.

Figure 2.1 Example of a camera tracker that uses the information obtained by tracking image patches. Reproduced with permission of the Oxford Metrics Group.

Figure 2.2 Match-moving examples for special effects and object placement in a dynamic scene. Top: smoke and other objects are added to a naval scene. Bottom: the rendering of a new building is added to an aerial view. Reproduced with permission of the Oxford Metrics Group.

Another technology based on video tracking and used by media production houses is motion capture. Motion capture systems are used to animate virtual characters from the tracked motion of real actors. Although markerless motion capture is receiving increasing attention, most motion-capture systems track a set of markers attached to an actor's body and limbs to estimate their poses (Figure 2.3). Specialised motion-capture systems recover the movements of real actors in 3D from the tracked markers. Then the motion of the makers is mapped onto characters generated by computer graphics.

Video tracking is also used for the analysis and the enhancement of sport events. As shown in the example of Figure 2.4 , a tracking algorithm can estimate the position of players in the field in order to gather statistics about a game (e.g. a football match). Statistics and enhanced visualisations aid the commentators, coaches and supporters in highlighting team tactics and player performance.

2.3 MEDICAL APPLICATIONS AND BIOLOGICAL RESEARCH

The motion-capture tools described in the previous section are also used for the analysis of human motion to improve the performance of athletes (Figure 2.5(a)-(b)) and for the analysis of the gait of a patient [3] to assess the condition of the joints and bones (Figure 2.5(c)). In general, video tracking has been increasingly used by medical systems to aid the diagnosis and to speed up the operator's task. For example, automated algorithms track the ventricular motion in ultrasound images [4-6]. Moreover, video tracking can estimate the position of particular soft tissues [7] or of instruments such as needles [8,9] and bronchoscopes [10] during surgery.

Figure 2.3 Examples of motion capture using a marker-based system. Left: retroreflective markers to be tracked by the system. Right: visualisation of the motion of a subject. Reproduced with permission of the Oxford Metrics Group.

In biological research, tracking the motion of non-human organisms allows one to analyse and to understand the effects of specific drugs or the effects of ageing [11-15]. Figure 2.6 shows two application examples where video tracking estimates the location of Escherichia coli bacteria and Caenorhabditis elegans worms.

Figure 2.4 Video tracking applied to media production and enhanced visualisation of sport events. Animations of the real scene can be generated from different view-points based on tracking data. Moreover, statistics regarding player positions are automatically gathered and may be presented as overlay or animation. Reproduced with permission of Mediapro.

Figure 2.5 Example of video tracking for medical and sport analysis applications. Motion capture is used to analyse the performance of a golfer (a), of a rower (b) and to analyse the gait of a patient (c). Reproduced with permission of the Oxford Metrics Group.

Figure 2.6 Examples of video tracking for medical research. Automated tracking of the position of Escherichia coli bacteria (left) and of Caenorhabditis elegans worms (right). Left: reproduced from [14]; right: courtesy of Gavriil Tsechpenakis, IUPUI.

2.4 SURVEILLANCE AND BUSINESS INTELLIGENCE

Video tracking is a desirable tool used in automated video surveillance for security, assisted living and business intelligence applications. In surveillance systems, tracking can be used either as a forensic tool or as a processing stage prior to algorithms that classify behaviours [16]. Moreover, video-tracking software combined with other video analytical tools can be used to redirect the attention of human operators towards events of interest.

Smart surveillance systems can be deployed in a variety of different indoor and outdoor environments such as roads, airports, ports, railway stations, public and private buildings (e.g. schools, banks and casinos). Examples of video surveillance systems (Figure 2.7) are the IBM Smart Surveillance System (S3) [17,18], the General Electrics VisioWave Intelligent Video Platform [19] and Object Video VEW [20,21].

Figure 2.7 Examples of object tracking in surveillance applications. (a)-(b): General Electric intelligent video platform; (c): ObjectVideo surveillance platform. The images are reproduced with permission of General Electric Company (a,b) and ObjectVideo (c).

Figure 2.8 Examples of video tracking for intelligent retail applications. Screen shots from IntelliVid software (American Dynamics).

Video tracking may also serve as an observation and measurement tool in retail environments (e.g. retail intelligence), such as supermarkets, where the position of customers is tracked over time [22] (Figure 2.8). Trajectory data combined with information from the point of sales (till) is used to build behavioural models describing where customers spend their time in the shop, how they interact with products dpending on their location, and what items they buy. By analysing this information, the marketing team can improve the product placement in the retail space. Moreover, gaze tracking in front of billboards can be used to automatically select the type of advertisment to show or to dynamically change its content based on the attention or the estimated marketing profile of a person, based for example on the estimated gender and age.

2.5 ROBOTICS AND UNMANNED VEHICLES

Another application area that extensively uses video-tracking algorithms is robotics. Robotic technology includes the development of humanoid robots, automated PTZ cameras and unmanned aerial vehicles (UAVs). Intelligent vision via one or more cameras mounted on the robots provide information that is used to interact with or navigate in the environment. Also environment exploration and mapping [23], as well as human-robot interaction via gesture recognition rely on video tracking [24].

The problem of estimating the global motion of robots and unmanned vehicles is related to the camera-tracking problem discussed in Section 2.2. While tracking algorithms for media production can be applied off-line, video trackers for robotics need to simultaneously localise in realtime the position of the robot (i.e. of the camera) and to generate a map of the environment. 3D localisation information is generated by tracking the position of prominent image features such as corners and edges [25,26], as shown in Figure 2.1.

Figure 2.9 Example of object tracking from an Unmanned Aerial Vehicle. Repreduced with permission of the Oxford Metrics Group.

Information on the 3D position is also used to generate a...