A Variational Approach to Video Registration with

Authors: Ravi Garg Anastasios Roussos Lourdes Agapito

Publish Date: 2013/04/02

Volume: 104, Issue: 3, Pages: 286-314

PDF Link

Abstract

This paper addresses the problem of nonrigid video registration or the computation of optical flow from a reference frame to each of the subsequent images in a sequence when the camera views deformable objects We exploit the high correlation between 2D trajectories of different points on the same nonrigid surface by assuming that the displacement of any point throughout the sequence can be expressed in a compact way as a linear combination of a lowrank motion basis This subspace constraint effectively acts as a trajectory regularization term leading to temporally consistent optical flow We formulate it as a robust soft constraint within a variational framework by penalizing flow fields that lie outside the lowrank manifold The resulting energy functional can be decoupled into the optimization of the brightness constancy and spatial regularization terms leading to an efficient optimization scheme Additionally we propose a novel optimization scheme for the case of vector valued images based on the dualization of the data term This allows us to extend our approach to deal with colour images which results in significant improvements on the registration results Finally we provide a new benchmark dataset based on motion capture data of a flag waving in the wind with dense ground truth optical flow for evaluation of multiframe optical flow algorithms for nonrigid surfaces Our experiments show that our proposed approach outperforms state of the art optical flow and dense nonrigid registration algorithmsOptical flow in the presence of nonrigid deformations is a challenging task and an important problem that continues to attract significant attention from the computer vision community It has wide ranging applications from medical imaging and video augmentation to nonrigid structure from motion Given a template image of a nonrigid object and an input image of it after deforming the task can be described as one of finding the displacement field warp that relates the input image back to the template In this paper we consider long video sequences instead of a single pair of frames—each of the images in the sequence must be aligned back to the reference frame Our work concerns the estimation of the vector field of displacements that maps pixels in the reference frame to each image in the sequence see Fig 1Video registration is equivalent to the problem of estimating dense optical flow varvecuvarvecxn between a reference frame I ref and each of the subsequent frames I n in a sequence We propose a multiframe optical flow algorithm that exploits temporal consistency by imposing subspace constraints on the 2D image trajectoriesThe strong correlation between 2D trajectories of different points on the same nonrigid surface can be exploited to impose temporal coherence by modelling long term temporal coherence imposing subspace constraints These trajectories lie on a lower dimensional manifold which leads to a significant reduction in the dimensionality of the problem while implicitly imposing some form of temporal smoothnessSubspace constraints have been used before both in the context of sparse point tracking Irani 2002 Brand 2001 Torresani et al 2001 Torresani and Bregler 2002 and optical flow Irani 2002 Garg et al 2010 in the rigid and nonrigid domains to allow correspondences to be obtained in low textured areas While Irani’s original rigid Irani 2002 formulation along with its nonrigid extensions Torresani et al 2001 Brand 2001 Torresani and Bregler 2002 relied on minimizing the linearized brightness constraint without smoothness priors Garg et al 2010 extended the subspace constraints to the continuous domain in the nonrigid case using a variational approach Nir et al 2008 propose a variational approach to optical flow estimation based on a spatiotemporal model However all of the above approaches impose the subspace constraint as a hard constraint Hard constraints are vulnerable to noise in the data and can be avoided by substituting them with principled robust constraintsIn this paper we extend the use of multiframe temporal smoothness constraints within a variational framework by providing a more principled energy formulation with a robust soft constraint which leads to improved results In practice we penalize deviations of the optical flow trajectories from the lowrank subspace manifold which acts as a temporal regularization term over long sequences We then take advantage of recent developments Chambolle 2004 Chambolle and Pock 2011 in variational methods and optimize the energy defining a variant of the dualitybased efficient numerical optimization scheme We are also able to prove that our soft constraint is preferable to a hard constraint imposed via reparameterization To do this we provide a formulation of the hard constraint and its optimization and we perform thorough experimental comparisons where we show that the results obtained via the soft constraint always outperform those obtained after reparameterizationThe paper is organized as follows In Sect 2 we describe related approaches and discuss the contributions of our work Section 3 defines the trajectory subspace constraints that we use in our formulation In Sect 4 we describe the energy and provide a discussion on the design of our effective trajectory regularizer Section 5 addresses the optimization of our proposed energy This is followed by a description of the estimation of the motion basis in Sect 6 In Sect 7 we propose the extension of our algorithm to vectorvalued images and Sect 8 discusses implementation details Finally Sect 9 describes the alternative formulation of the subspace constraint as a hard constraint while Sect 10 describes our experimental evaluationVariational methods formulate the optical flow or image alignment problems as the optimization of an energy functional in a continuous domain Stemming from Horn and Schunck’s original approach Horn and Schunck 1981 the energy incorporates a data term that accounts for the brightness constancy assumption and a regularization term that allows to fillin flow information in low textured areas Variational methods have seen a huge surge in recent years due to the development of more sophisticated and robust data fidelity terms which are robust to changes in image brightness or occlusions Brox and Malik 2011 Brox et al 2004 the addition of efficient regularization terms such as Total Variation TV Zach et al 2007 Wedel et al 2008 or temporal smoothing terms Weickert and Schnörr 2001b and new optimization strategies that allow computation of highly accurate Wedel et al 2009 and real time optical flow Zach et al 2007 even in the presence of large displacements Alvarez et al 2000 Brox and Malik 2011 Steinbruecker et al 2009One important recent advance in variational optical flow methods has been the development of the duality based efficient optimization of the socalled TVmathbfL1 formulation Zach et al 2007 Chambolle and Pock 2011 which owes its name to the Total Variation that is used for regularization and the robust mathbfL1norm that is used in the data fidelity term An example of this class is the Improved TVmathbfL1 ITVmathbfL1 method Wedel et al 2009 which yielded notable quantitative performance by also carefully considering some practical aspects of the optical flow algorithmDuplication of the optimization variable via a quadratic relaxation is used to decouple the linearized data and regularization terms decomposing the optimization problem into two each of which is a convex energy that can be solved in a globally optimal manner The minimization algorithm then alternates between solving for each of the two variables assuming the other one fixed One of the key advantages of this decoupling scheme is that since the data term is pointwise independent its optimization can be highly parallelized using graphics hardware Zach et al 2007 Following its success in optical flow computation this optimization scheme has since been successfully applied to motion and disparity estimation Pock et al 2010 and real time dense 3D reconstruction Newcombe et al 2011 Stuehmer et al 2010 In this work we adopt this efficient duality based TVmathbfL1 optimization scheme Zach et al 2007 and extend it to the case of multiframe optical flow for video registration by modelling long term temporal coherence imposing subspace constraints

Keywords: