Feature Extraction in Regression and Classification with Structured Predictors

A typical task in statistical modeling is variable selection. In this thesis, however, not only variable selection but feature extraction is investigated. Feature extraction goes beyond variable selection in the sense that not only variables are selected but features which depend on the special nature of the data considered. In this dissertation, variables with a special structure are considered and used as predictors in regression and classification problems. High-dimensional signal-like (metric) covariates are the first type of data investigated. A typical example for this kind of data are functional predictors in signal regression, which can only be observed at (a high number of) distinct measurement points but are realizations of (more or less) smooth functions. In this case, feature extraction can be defined as 'the identification of relevant parts of the signal'. For that purpose, a Boosting technique is developed, which can also be applied to curves of protein intensities obtained from mass spectrometry in proteomics. Simulation studies and real world data applications show that the proposed procedure is a highly competitive alternative to existing approaches. Categorical covariates, which are usually dummy-coded and hence result in groups of dummy variables, are another very interesting type of structured regressors. If predictors are ordinal, however, the levels' ordering is typically ignored in regression modeling, or methods for metric covariates are (wrongly) applied. In this thesis, penalized likelihood methods are proposed which take the ordinal scale level into account using a difference penalty on adjacent dummy coefficients. Besides variable selection, the identification of relevant differences between categories of both ordinal and nominal predictors is considered, and appropriate L1-type regularization techniques are presented. The methods are investigated from a practical and a theoretical point of view. It is shown that the proposed procedures perform quite well, also in comparison with alternative approaches. Categorical covariates serving as (potentially) effect modifying factors in varying-coefficient models are considered, too. Finally, approaches for nonparametric feature extraction using nearest neighbor methods are presented. The performance of the proposed nearest neighbor ensemble technique is quite encouraging.