Morethanadecadeago,combiningmultipleclassi?erswasproposedasap- siblesolutiontotheproblemsposedbythetraditionalpatternclassi?cation approachwhichinvolvedselectingthebestclassi?erfromasetofcandidates basedontheirexperimentalevaluation. Asnoclassi?erisknowntobethebest forallcasesandtheselectionofthebestclassi?erforagivenpracticaltaskis verydi?cult,diverseresearchcommunities,includingMachineLearning,N- ralNetworks,PatternRecognition,andStatistics,addressedtheengineering problemofhowtoexploitthestrengthswhileavoidingtheweaknessesofd- ferentdesigns. Thisambitiousresearchtrendwasalsomotivatedbyempirical observationsaboutthecomplementarityofdi?erentclassi?erdesigns,natural requirementsofinformationfusionapplications,andintrinsicdi?cultiesasso- atedwiththeoptimalchoiceofsomeclassi?erdesignparameters,suchasthe architectureandtheinitialweightsforaneuralnetwork. Afteryearsofresearch, thecombinationofmultipleclassi?ershasbecomeawellestablishedandexciting researcharea,whichprovidese?ectivesolutionstodi?cultpatternrecognition problems. Aconsiderablebodyofempiricalevidencesupportsthemeritof- signingcombinedsystemswhoseaccuracyishigherthanthatofeachindividual classi?er,andvariousmethodsforthegenerationandthecombinationofm- tipleclassi?
ershavebecomeavailable. However,despitetheprovedutilityof multipleclassi?ersystems,nogeneralanswertotheoriginalquestionaboutthe possibilityofexploitingthestrengthswhileavoidingtheweaknessesofdi?erent classi?erdesignshasyetemerged. Otherfundamentalissuesarealsoamatterof on-goingresearchindi?erentresearchcommunities. Theresultsachievedd- ingthepastyearsarealsospreadoverdi?erentresearchcommunities,andthis makesitdi?culttoexchangesuchresultsandpromotetheircross-fertilization. Theacknowledgmentofthefundamentalrolethatthecreationofacommon internationalforumforresearchersofthediversecommunitiescouldplayfor theadvancementofthisresearch?eldmotivatedthepresentseriesofwo- shopsonmultipleclassi?ersystems. Followingitspredecessors,MultipleCl- si?erSystems2000(SpringerISBN3-540-67704-6)and2001(SpringerISBN 3-540-42284-6),thisvolumecontainstheproceedingsoftheThirdInternational WorkshoponMultipleClassi?erSystems(MCS2002),heldattheGrandHotel ChiaLaguna,Cagliari,Italy,onJune24-26,2002. The29papersselectedby thescienti?ccommitteehavebeenorganizedinsessionsdealingwithbagging andboosting,ensemblelearningandneuralnetworks,combinationstrategies, designmethodologies,analysisandperformanceevaluation,andapplications.
Theworkshopprogramandthisvolumeareenrichedwiththreeinvitedtalks givenbyJoydeepGhosh(UniversityofTexas,USA),TrevorHastie(Stanford University,USA),andSarunasRaudys(VilniusGediminasTechnicalUniversity, Lithuania). Papersweresubmittedfromresearchersofthefourdiversecom- nities,socon?rmingthatthisseriesofworkshopscanbecomeacommonforum VI Foreword forexchangingviewsandreportinglatestresearchresults. Asfortheprevious editions,thesigni?cantnumberofpapersdealingwithrealpatternrecognition applicationsareproofofthepracticalutilityofmultipleclassi?ersystems. This workshopwassupportedbytheUniversityofCagliari,Italy,theUniversityof Surrey,Guildford,UnitedKingdom,andtheDepartmentofElectricalandEl- tronicEngineeringoftheUniversityofCagliari. Allthesesupportsaregratefully acknowledged. WealsothanktheInternationalAssociationforPatternRecog- tionanditsTechnicalCommitteeTC1onStatisticalPatternRecognitionTe- niquesforsponsoringMCS2002. Wewishtoexpressourappreciationtoallthose whohelpedtoorganizeMCS2002. Firstofall,wewouldliketothankallthe membersoftheScienti?cCommitteewhoseprofessionalismwasinstrumental increatingaveryinterestingtechnicalprogram.
Specialthanksareduetothe membersoftheOrganizingCommittee,GiorgioFumera,GiorgioGiacinto,and GianLucaMarcialisfortheirindispensablecontributionstotheMCS2002web sitemanagement,localorganization,andproceedingspreparation. April2002 FabioRoliandJosefKittler WorkshopChairs F. Roli(Univ. ofCagliari,Italy) J. Kittler(Univ. ofSurrey,UnitedKingdom) Scienti?cCommittee J. A. Benediktsson(Iceland) M. Kamel(Canada) H. Bunke(Switzerland) L. I. Kuncheva(UK) L. P. Cordella(Italy) L. Lam(HongKong) B. V. Dasarathy(USA) D. Landgrebe(USA) R. P. W. Duin(TheNetherlands) Dar-ShyangLee(USA) C. Furlanello(Italy) D. Partridge(UK) J. Ghosh(USA) A. J. C. Sharkey(UK) T. K. Ho(USA) K. Tumer(USA) S. Impedovo(Italy) G. Vernazza(Italy) N. Intrator(Israel) T. Windeatt(UK) A. K. Jain(USA) LocalCommittee G. Fumera(Univ. ofCagliari,Italy) G. Giacinto(Univ. ofCagliari,Italy) G. L. Marcialis(Univ. ofCagliari,Italy) Organizedby Dept. ofElectricalandElectronicEngineeringoftheUniversityofCagliari UniversityofSurrey Sponsoredby UniversityofCagliari UniversityofSurrey Dept. ofElectricalandElectronicEngineeringoftheUniversityofCagliari TheInternationalAssociationforPatternRecognition Supportedby UniversityofCagliari Dept. ofElectricalandElectronicEngineeringoftheUniversityofCagliari UniversityofSurrey TableofContents InvitedPapers Multiclassi?
erSystems:BacktotheFuture...1 J. Ghosh SupportVectorMachines,KernelLogisticRegressionandBoosting...16 J. Zhu,T. Hastie MultipleClassi?cationSystemsintheContextofFeatureExtractionand Selection...27 ? S. Raudys BaggingandBoosting BoostedTreeEnsemblesforSolvingMulticlassProblems...42 T. Windeatt,G. Ardeshir DistributedPastingofSmallVotes...52 N. V. Chawla,L. O. Hall,K. W. Bowyer,T. E. Moore,Jr. , W. P. Kegelmeyer BaggingandBoostingfortheNearestMeanClassi?er:E?ectsofSample SizeonDiversityandAccuracy...62 M. Skurichina,L. I. Kuncheva,R. P. W. Duin HighlightingHardPatternsviaAdaboostWeightsEvolution ...72 B. Caprile,C. Furlanello,S. Merler UsingDiversitywithThreeVariantsofBoosting:Aggressive,Conservative, andInverse ...81 L. I. Kuncheva,C. J. Whitaker EnsembleLearningandNeuralNetworks MultistageNeuralNetworkEnsembles...91 S. Yang,A. Browne,P. D. Picton ForwardandBackwardSelectioninRegressionHybridNetwork...98 S. Cohen,N. Intrator TypesofMultinetSystem...108 A. J. C.
Invited Papers.- Multiclassifier Systems: Back to the Future.- Support Vector Machines, Kernel Logistic Regression and Boosting.- Multiple Classification Systems in the Context of Feature Extraction and Selection.- Bagging and Boosting.- Boosted Tree Ensembles for Solving Multiclass Problems.- Distributed Pasting of Small Votes.- Bagging and Boosting for the Nearest Mean Classifier: Effects of Sample Size on Diversity and Accuracy.- Highlighting Hard Patterns via AdaBoost Weights Evolution.- Using Diversity with Three Variants of Boosting: Aggressive, Conservative, and Inverse.- Ensemble Learning and Neural Networks.- Multistage Neural Network Ensembles.- Forward and Backward Selection in Regression Hybrid Network.- Types of Multinet System.- Discriminant Analysis and Factorial Multiple Splits in Recursive Partitioning for Data Mining.- Design Methodologies.- New Measure of Classifier Dependency in Multiple Classifier Systems.- A Discussion on the Classifier Projection Space for Classifier Combining.- On the General Application of the Tomographic Classifier Fusion Methodology.- Post-processing of Classifier Outputs in Multiple Classifier Systems.- Combination Strategies.- Trainable Multiple Classifier Schemes for Handwritten Character Recognition.- Generating Classifier Ensembles from Multiple Prototypes and Its Application to Handwriting Recognition.- Adaptive Feature Spaces for Land Cover Classification with Limited Ground Truth Data.- Stacking with Multi-response Model Trees.- On Combining One-Class Classifiers for Image Database Retrieval.- Analysis and Performance Evaluation.- Bias-Variance Analysis and Ensembles of SVM.- An Experimental Comparison of Fixed and Trained Fusion Rules for Crisp Classifier Outputs.- Reduction of the Boasting Bias of Linear Experts.-Analysis of Linear and Order Statistics Combiners for Fusion of Imbalanced Classifiers.- Applications.- Boosting and Classification of Electronic Nose Data.- Content-Based Classification of Digital Photos.- Classifier Combination for In Vivo Magnetic Resonance Spectra of Brain Tumours.- Combining Classifiers of Pesticides Toxicity through a Neuro-fuzzy Approach.- A Multi-expert System for Movie Segmentation.- Decision Level Fusion of Intramodal Personal Identity Verification Experts.- An Experimental Comparison of Classifier Fusion Rules for Multimodal Personal Identity Verification Systems.
