Penalized Versus Constrained Generalized Eigenvalue Problems

<p>We investigate the difference between using an ℓ<sub>1</sub> penalty versus an ℓ<sub>1</sub> constraint in generalized eigenvalue problems arising in multivariate analysis. Our main finding is that the ℓ<sub>1</sub> penalty may fail to provide very sparse solutions; a severe disadvantage for variable selection that can be remedied by using an ℓ<sub>1</sub> constraint. Our claims are supported both by empirical evidence and theoretical analysis. Finally, we illustrate the advantages of the ℓ<sub>1</sub> constraint in the context of discriminant analysis and principal component analysis. Supplementary materials for this article are available online.</p>