Joint covariate selection and joint subspace selection for multiple classification problems

Thumbnail Image
Penn collection
Lab Papers (GRASP)
Degree type
Variable selection
Subspace selection
Group Lasso
Regularization path
Supervised dimensionality reduction
Multitask learning
Block norm
Trace norm
Random projections
Grant number
Copyright date
Related resources
Obozinski, Guillaume
Jordan, Michael I

We address the problem of recovering a common set of covariates that are relevant simultaneously to several classification problems. By penalizing the sum of ℓ2-norms of the blocks of coefficients associated with each covariate across different classification problems, similar sparsity patterns in all models are encouraged. To take computational advantage of the sparsity of solutions at high regularization levels, we propose a blockwise path-following scheme that approximately traces the regularization path. As the regularization coefficient decreases, the algorithm maintains and updates concurrently a growing set of covariates that are simultaneously active for all problems. We also show how to use random projections to extend this approach to the problem of joint subspace selection, where multiple predictors are found in a common low-dimensional subspace. We present theoretical results showing that this random projection approach converges to the solution yielded by trace-norm regularization. Finally, we present a variety of experimental results exploring joint covariate selection and joint subspace selection, comparing the path-following approach to competing algorithms in terms of prediction accuracy and running time.

Date Range for Data Collection (Start Date)
Date Range for Data Collection (End Date)
Digital Object Identifier
Series name and number
Publication date
Journal title
Volume number
Issue number
Publisher DOI
Journal Issue
Postprint version. Published in: Joint covariate selection and joint subspace selection for multiple classification problems, G. Obozinski, B.Taskar, M.I. Jordan, Statistics and Computing, Jan. 2009. The original publication is available at Publisher URL:
Recommended citation