Optimization Algorithms on Matrix Manifolds

List of Algorithms xi
Foreword, by Paul Van Dooren xiii
Notation Conventions xv


Chapter 1. Introduction 1
Chapter 2. Motivation and Applications 5
2.1 A case study: the eigenvalue problem 5
2.1.1 The eigenvalue problem as an optimization problem 7
2.1.2 Some benefits of an optimization framework 9
2.2 Research problems 10
2.2.1 Singular value problem 10
2.2.2 Matrix approximations 12
2.2.3 Independent component analysis 13
2.2.4 Pose estimation and motion recovery 14
2.3 Notes and references 16


Chapter 3. Matrix Manifolds: First-Order Geometry 17
3.1 Manifolds 18
3.1.1 Definitions: charts, atlases, manifolds 18
3.1.2 The topology of a manifold* 20
3.1.3 How to recognize a manifold 21
3.1.4 Vector spaces as manifolds 22
3.1.5 The manifolds R^{n x p} and R^{n x p} 22
3.1.6 Product manifolds 23
3.2 Differentiable functions 24
3.2.1 Immersions and submersions 24
3.3 Embedded submanifolds 25
3.3.1 General theory 25
3.3.2 The Stiefel manifold 26
3.4 Quotient manifolds 27
3.4.1 Theory of quotient manifolds 27
3.4.2 Functions on quotient manifolds 29
3.4.3 The real projective space RP^{n x 1} 30
3.4.4 The Grassmann manifold Grass(p, n) 30
3.5 Tangent vectors and differential maps 32
3.5.1 Tangent vectors 33
3.5.2 Tangent vectors to a vector space 35
3.5.3 Tangent bundle 36
3.5.4 Vector fields 36
3.5.5 Tangent vectors as derivations? 37
3.5.6 Differential of a mapping 38
3.5.7 Tangent vectors to embedded submanifolds 39
3.5.8 Tangent vectors to quotient manifolds 42
3.6 Riemannian metric, distance, and gradients 45
3.6.1 Riemannian submanifolds 47
3.6.2 Riemannian quotient manifolds 48
3.7 Notes and references 51


Chapter 4. Line-Search Algorithms on Manifolds 54
4.1 Retractions 54
4.1.1 Retractions on embedded submanifolds 56
4.1.2 Retractions on quotient manifolds 59
4.1.3 Retractions and local coordinates* 61
4.2 Line-search methods 62
4.3 Convergence analysis 63
4.3.1 Convergence on manifolds 63
4.3.2 A topological curiosity* 64
4.3.3 Convergence of line-search methods 65
4.4 Stability of fixed points 66
4.5 Speed of convergence 68
4.5.1 Order of convergence 68
4.5.2 Rate of convergence of line-search methods* 70
4.6 Rayleigh quotient minimization on the sphere 73
4.6.1 Cost function and gradient calculation 74
4.6.2 Critical points of the Rayleigh quotient 74
4.6.3 Armijo line search 76
4.6.4 Exact line search 78
4.6.5 Accelerated line search: locally optimal conjugate gradient 78
4.6.6 Links with the power method and inverse iteration 78
4.7 Refining eigenvector estimates 80
4.8 Brockett cost function on the Stiefel manifold 80
4.8.1 Cost function and search direction 80
4.8.2 Critical points 81
4.9 Rayleigh quotient minimization on the Grassmann manifold 83
4.9.1 Cost function and gradient calculation 83
4.9.2 Line-search algorithm 85
4.10 Notes and references 86


Chapter 5. Matrix Manifolds: Second-Order Geometry 91
5.1 Newton's method in Rⁿ 91
5.2 Affine connections 93
5.3 Riemannian connection 96
5.3.1 Symmetric connections 96
5.3.2 Definition of the Riemannian connection 97
5.3.3 Riemannian connection on Riemannian submanifolds 98
5.3.4 Riemannian connection on quotient manifolds 100
5.4 Geodesics, exponential mapping, and parallel translation 101
5.5 Riemannian Hessian operator 104
5.6 Second covariant derivative* 108
5.7 Notes and references 110


Chapter 6. Newton's Method 111
6.1 Newton's method on manifolds 111
6.2 Riemannian Newton method for real-valued functions 113
6.3 Local convergence 114
6.3.1 Calculus approach to local convergence analysis 117
6.4 Rayleigh quotient algorithms 118
6.4.1 Rayleigh quotient on the sphere 118
6.4.2 Rayleigh quotient on the Grassmann manifold 120
6.4.3 Generalized eigenvalue problem 121
6.4.4 The nonsymmetric eigenvalue problem 125
6.4.5 Newton with subspace acceleration: Jacobi-Davidson 126
6.5 Analysis of Rayleigh quotient algorithms 128
6.5.1 Convergence analysis 128
6.5.2 Numerical implementation 129
6.6 Notes and references 131


Chapter 7. Trust-Region Methods 136
7.1 Models 137
7.1.1 Models in Rⁿ 137
7.1.2 Models in general Euclidean spaces 137
7.1.3 Models on Riemannian manifolds 138
7.2 Trust-region methods 140
7.2.1 Trust-region methods in Rⁿ 140
7.2.2 Trust-region methods on Riemannian manifolds 140
7.3 Computing a trust-region step 141
7.3.1 Computing a nearly exact solution 142
7.3.2 Improving on the Cauchy point 143
7.4 Convergence analysis 145
7.4.1 Global convergence 145
7.4.2 Local convergence 152
7.4.3 Discussion 158
7.5 Applications 159
7.5.1 Checklist 159
7.5.2 Symmetric eigenvalue decomposition 160
7.5.3 Computing an extreme eigenspace 161
7.6 Notes and references 165


Chapter 8. A Constellation of Superlinear Algorithms 168
8.1 Vector transport 168
8.1.1 Vector transport and affine connections 170
8.1.2 Vector transport by differentiated retraction 172
8.1.3 Vector transport on Riemannian submanifolds 174
8.1.4 Vector transport on quotient manifolds 174
8.2 Approximate Newton methods 175
8.2.1 Finite difference approximations 176
8.2.2 Secant methods 178
8.3 Conjugate gradients 180
8.3.1 Application: Rayleigh quotient minimization 183
8.4 Least-square methods 184
8.4.1 Gauss-Newton methods 186
8.4.2 Levenberg-Marquardt methods 187
8.5 Notes and references 188


A. Elements of Linear Algebra, Topology, and Calculus 189
A.1 Linear algebra 189
A.2 Topology 191
A.3 Functions 193
A.4 Asymptotic notation 194
A.5 Derivatives 195
A.6 Taylor's formula 198


Bibliography 201
Index 221