support vector and kernel learning

... important and active field of all Machine Learning research. zSpecial issues of Machine Learning Journal, and Journal of Machine Learning Research.z Kernel Machines: large class of learning ... ModularityzAny kernel -based learning algorithm composed of two modules:–A general purpose learning machine–A problem specific kernel functionzAny K-B algorithm can be fitted with any kernel zKernels ... ),,=+=+∑α Support Vector and Kernel MachinesLimitations ofLLMsLinear classifiers cannot deal withzNon-linearly separable datazNoisy dataz+ this formulation only deals with vectorial...

... kernel ),(),(),(),(2),(2),(),(exp),(),(exp),()),((),(1112111211zzKxxKzxKzxKzxKzzKxxKzxKzxKzxKczxKzxKd=−+−==+=σσwww .support- vector. netMaking kernels !From features: start from the features, then obtain the kernel. Example: the polynomial kernel, the string kernel, …19www .support- vector. net Learning ... ,())=+∑αφ φ1 Support Vector and Kernel Methods for Pattern Recognitionwww .support- vector. netA Little History! Support Vector Machines (SVM) introduced in COLT-92 (conference on learning theory) ... (gaussian kernels)17www .support- vector. netMaking kernels!From kernels (see closure properties):can obtain complex kernels by combining simpler ones according to specific ruleswww .support- vector. netClosure...

... ACL 2007 Demo and Poster Sessions, pages 57–60,Prague, June 2007.c2007 Association for Computational Linguistics Support Vector Machines for Query-focused Summarization trained and evaluated ... (which happens in2The mean, median, standard deviation and histogram of theoverlapping distribution are calculated and included as features.58this case), and a simple iterative procedure calledMapping-Convergence ... text in theSCU label and all its contributors is stemmed and stop words are removed, obtaining a set of stemvectors for each SCU. The system summary text isalso stemmed and freed from stop words....

... The resulting vocabu-lary consisted of 276 words and 56 POS tags.4.3 Support Vector Machines Support vector machines (SVMs) are a machine learning technique used in a variety of text classi-fication ... e.g. Lee and Myaeng’s (2002) genre and subject detection work and Boulis and Ostendorf’s (2005) work on featureselection for topic classification.For our LM classifiers, we followed Boulis and Ostendorf’s ... Computational LinguisticsReading Level Assessment Using Support Vector Machines and Statistical Language ModelsSarah E. SchwarmDept. of Computer Science and EngineeringUniversity of WashingtonSeattle,...

... support vector learning for chunk identification. In Proceed-ings of the 4th Conference on CoNLL-2000 and LLL-2000, pages 142–144.Taku Kudo and Yuji Matsumoto. 2001. Chunking with support vector ... of support vec-tor machines using sequential minimal optimization.In Bernhard Sch¨olkopf, Christopher J.C. Burges, and AlexanderJ. Smola, editors, Advances in Kernel Meth-ods: Support Vector ... pool-based active learning where. To train an SVM is to findthe and the by solving the following optimiza-tion problem:maximizesubject to3 Active Learning for Support Vector Machines3.1...

... support vector onlyneedtodescribethedatawithknowncategory,thendomainclassifer(SVDC),thenanincremental learning obtainingthedescriptionboundaryofthisclassofdata.algorithmbasedonSVDCwasproposed.ThebasicideaofFinally,wecanclassifytheunknownbinary-classdatathisincrementalalgorithmistoobtaintheinitialtargetaccordingtotheobtainedboundary.conceptsusingSVDCduringthetrainingprocedure and thenInthispaperourincremental learning algorithmisbasedupdatethesetargetconceptsbyanupdatingmodel.DifierentonSVDC, and thisalgorithmismotivatedbythefromtheexistedincremental learning approaches,inourperson -learning procedure.When learning acomplicatedalgorithm,themodelupdatingprocedureequalstosolveaconcept,peopleusuallyobtainainitialconceptbyusingpartquadraticprogramming(QP)problem, and theupdatedmodelofusefulinformation,thenupdatetheobtainedconceptbystillownsthepropertyofsparssolution.Comparedwithotherutilizingnewinformation.Intermofourincrementalexistedincrementallearningalgorithms,theinverseprocedurealgorithmbasedonSVDC,itfirstlyutilizepartofdataofouralgorithm(i.e.decreasing learning) iseasytoconduct(memoryspacepermitting),thenobtainaconcept(namelythewithoutextracomputation.Experimentresultsshowourparameterofobtaineddecisionhypersurface)bySVDCalgorithmiseffectiveandfeasible. learning algorithm,finallyaccordingtotheinformationofdecisionhypersurfaceacquiredinlaststep,updatetheparameterofdecisionhypersurfacegainedinlaststeputilizingKeywords: Support Vector Machines, Support Vector DomainspecializedupdatingmodelintheprocessofincrementalClassifier,Incremental learning, Classification. learning, namelyupdatingtheknownconcept.Ouralgorithmownsthefollowingcharacters:1.INTRODUCTION1)TheincrementalupdatingmodelinthisalgorithmWithlargeamountsofdataavailabletomachine learning hasasimilarmathematicsformcomparedwithcommunity,theneedtodesigntechniquesthatscalewellisstandardSVDCalgorithm, and anyalgorithmusedmorecriticalthanbefore.AssomedatamaybecollectedovertoobtainthestandardSVDCcanalsobeusedtolongperiods,thereisalsoacontinuousneedtoincorporatetheobtaintheupdatingmodelofouralgorithm;newdataintothepreviouslylearnedconcept.Incremental2)Theinverseprocedureofthisalgorithm,i.e.the learning techniquescansatisfytheneedforboththescalabilitydecreasing learning procedureiseasyto and incrementalupdate.implement,thatistosaywhenweperceivethe Support vector machine(SVM)isbasedonstatisticalgeneralizationperformancedroppedinthe learning theory,whichhasdevelopedoverlastthreedecadesincrementalprocess,wecaneasilyreturnlaststep[1,2].Ithasbeenprovenverysuccessfulinmanyapplicationswithoutextracomputation;[3,4,5,6].SVMisasupervisedbinary-classclassifier,whenTheexperimentalresultsshowthe learning performancewetrainsamplesusingSVM,thecategoriesofthesamplesareofthisalgorithmapproachesthatofbatchtraining, and neededtobeknown.However,inmanycases,itisrarethatperformancewellinlarge-scaledatasetcomparedtootherwecanobtainthedatawiththeircategorybeknown,inotherSVDCincremental learning algorithm.words,mostoftheobtaineddata'scategoriesareunknown.InTherestofthispaperisorganizedasfollows.Insection2thissituation,traditionalSVMisn'tappropriate.wegiveanintroductionofSVDC, and insection3wepresentTAXetalproposedamethodfordatadomaindescriptionourincrementalalgorithm.Experimental and resultscalled support vector domaindescription(SVDD)[7], and itisconcerningtheproposedalgorithmareofferedinSection4.usedtodescribedatadomain and deleteoutliers.ThekeyideaSection5collectsthemainconclusions.OfSVDDistodescribeoneclassofdatabyfindingaspherewithminimumvolume,whichcontainsthisclassofdata.Proc.5thIEEEInt.Conf.onCognitiveInformatics(ICCI'06)Y.Y.Yao,Z.Z.Shi,Y.Wang, and W.Kinsner(Eds.)801-4244-0475-4/06/$20.OO@2006IEEE802. Support Vector DomainClassifierwithconstrains,==1, and 0<a,<C.Wherethe2.1 Support Vector DomainDescription[7]innerproducthasbeenreplacedwith kernel functionK(.,.), and K(.,.)isadefinite kernel satisfyingmercerOfadatasetcontaiingNdataobjcondition,forexampleapopularchoiceistheGaussianOfadatasetcontainingNdataobjects,enl(,)=ep-xz2/2),>0fx,Z=1, ... support vector onlyneedtodescribethedatawithknowncategory,thendomainclassifer(SVDC),thenanincremental learning obtainingthedescriptionboundaryofthisclassofdata.algorithmbasedonSVDCwasproposed.ThebasicideaofFinally,wecanclassifytheunknownbinary-classdatathisincrementalalgorithmistoobtaintheinitialtargetaccordingtotheobtainedboundary.conceptsusingSVDCduringthetrainingprocedure and thenInthispaperourincremental learning algorithmisbasedupdatethesetargetconceptsbyanupdatingmodel.DifierentonSVDC, and thisalgorithmismotivatedbythefromtheexistedincremental learning approaches,inourperson -learning procedure.When learning acomplicatedalgorithm,themodelupdatingprocedureequalstosolveaconcept,peopleusuallyobtainainitialconceptbyusingpartquadraticprogramming(QP)problem, and theupdatedmodelofusefulinformation,thenupdatetheobtainedconceptbystillownsthepropertyofsparssolution.Comparedwithotherutilizingnewinformation.Intermofourincrementalexistedincrementallearningalgorithms,theinverseprocedurealgorithmbasedonSVDC,itfirstlyutilizepartofdataofouralgorithm(i.e.decreasing learning) iseasytoconduct(memoryspacepermitting),thenobtainaconcept(namelythewithoutextracomputation.Experimentresultsshowourparameterofobtaineddecisionhypersurface)bySVDCalgorithmiseffectiveandfeasible. learning algorithm,finallyaccordingtotheinformationofdecisionhypersurfaceacquiredinlaststep,updatetheparameterofdecisionhypersurfacegainedinlaststeputilizingKeywords: Support Vector Machines, Support Vector DomainspecializedupdatingmodelintheprocessofincrementalClassifier,Incremental learning, Classification. learning, namelyupdatingtheknownconcept.Ouralgorithmownsthefollowingcharacters:1.INTRODUCTION1)TheincrementalupdatingmodelinthisalgorithmWithlargeamountsofdataavailabletomachine learning hasasimilarmathematicsformcomparedwithcommunity,theneedtodesigntechniquesthatscalewellisstandardSVDCalgorithm, and anyalgorithmusedmorecriticalthanbefore.AssomedatamaybecollectedovertoobtainthestandardSVDCcanalsobeusedtolongperiods,thereisalsoacontinuousneedtoincorporatetheobtaintheupdatingmodelofouralgorithm;newdataintothepreviouslylearnedconcept.Incremental2)Theinverseprocedureofthisalgorithm,i.e.the learning techniquescansatisfytheneedforboththescalabilitydecreasing learning procedureiseasyto and incrementalupdate.implement,thatistosaywhenweperceivethe Support vector machine(SVM)isbasedonstatisticalgeneralizationperformancedroppedinthe learning theory,whichhasdevelopedoverlastthreedecadesincrementalprocess,wecaneasilyreturnlaststep[1,2].Ithasbeenprovenverysuccessfulinmanyapplicationswithoutextracomputation;[3,4,5,6].SVMisasupervisedbinary-classclassifier,whenTheexperimentalresultsshowthe learning performancewetrainsamplesusingSVM,thecategoriesofthesamplesareofthisalgorithmapproachesthatofbatchtraining, and neededtobeknown.However,inmanycases,itisrarethatperformancewellinlarge-scaledatasetcomparedtootherwecanobtainthedatawiththeircategorybeknown,inotherSVDCincremental learning algorithm.words,mostoftheobtaineddata'scategoriesareunknown.InTherestofthispaperisorganizedasfollows.Insection2thissituation,traditionalSVMisn'tappropriate.wegiveanintroductionofSVDC, and insection3wepresentTAXetalproposedamethodfordatadomaindescriptionourincrementalalgorithm.Experimental and resultscalled support vector domaindescription(SVDD)[7], and itisconcerningtheproposedalgorithmareofferedinSection4.usedtodescribedatadomain and deleteoutliers.ThekeyideaSection5collectsthemainconclusions.OfSVDDistodescribeoneclassofdatabyfindingaspherewithminimumvolume,whichcontainsthisclassofdata.Proc.5thIEEEInt.Conf.onCognitiveInformatics(ICCI'06)Y.Y.Yao,Z.Z.Shi,Y.Wang, and W.Kinsner(Eds.)801-4244-0475-4/06/$20.OO@2006IEEE802. Support Vector DomainClassifierwithconstrains,==1, and 0<a,<C.Wherethe2.1 Support Vector DomainDescription[7]innerproducthasbeenreplacedwith kernel functionK(.,.), and K(.,.)isadefinite kernel satisfyingmercerOfadatasetcontaiingNdataobjcondition,forexampleapopularchoiceistheGaussianOfadatasetcontainingNdataobjects,enl(,)=ep-xz2/2),>0fx,Z=1, ... ,~NJ}adescriptioniSrequired.Wetrytofindakre:Kxz=pJ1X_12221a>.{xs,ind1.,}acdscprequreeWwtrtindmaTodeterminewhetheratestpointiszwithintheclosed and compactsphereareaQwithminimumsphere,thedistancetothecenterofthespherehastobevolume,whichcontainall(ormostof)theneededobjectscalculated.AtestobjectzacceptedwhenthisdistanceisQ, and theoutliersareoutsideQ.Figure1showsthesmallthantheradius,i.e.,when(z-a)T(z-a)<R2.sketchof Support Vector DomainDescription(SVDD).Expressingthecenterofthesphereintermofthe support support vector vector,weacceptobjectswhenZ-a2=K(z,z) 2aiK(xz)+ZEaiaK(x1,xj)R2ijoutliersag6jc(5+cassiicationbo.urdary2.2 Support Vector DomainClassifier0O'*++...

... 1043004005006007008009001000Number of SamplesNumber of Support Vectors300 400 500 600 700 800 900 100000.511.522.533.544.55Number of Support VectorsTime (hours)0 0.5 1 1.5 2 2.5 3 3.5 4...

... Σ. . .output σ (Σ υi k (x,xi))weightsυ1υ2 υm. . .. . .test vector x support vectors x1 xnmapped vectors Φ(xi), Φ(x)Φ(x)Φ(xn)dot product (Φ(x).Φ(xi)) = k (x,xi)(...