The essentials of AB and multivariate testing

A B THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG www i nv espcr o com CONTENT I nt r oduct i on 01 Whati sA/ Bt est i ng( spl i tt est i ng) ? 03 Whati sMul t i v ar i at eTest i ng? 11 TheDi f f er enceBet weenA/ BTest i ngandMul t i v ar i at eTest i ng 18 Compar i ngMul t i v ar i at e&ABTest i ngTool s( updat edf or2018) 23 ThePr ocessofCr eat i ngaSuccessf ul A/ BTest 29 WhatEl ement sShoul dYouI ncl udei nanA/ BTest ? 34 Mul t i v ar i at eandA/ BTest i ngBestPr act i ces 54 16mi st ak est hatwi l l k i l l y ourA/ Bt est i ng ( andwhaty oucandoaboutt hem) 61 I NTRODUCTI ON Mostwebsi t escont i nuet ost r uggl ewi t hi ncr easi ngt hei ronl i neconv er si onr at es Onav er age, anecommer cest or econv er t sar ound3%ofi t st r af fici nt ocust omer s Ther emai ni ng97% oft hev i si t or sl eav et hewebsi t ewi t houtev erpl aci nganor der Ot hert y pesofwebsi t esst r uggl ewi t hconv er si onaswel l Onav er age, al eadgener at i onorsubscr i pt i on websi t esconv er tar ound15%ofi t st r af fici nt ocust omer s Ther emai ni ng85%l eav et hewebsi t ewi t hout ev erpl aci nganor der I fy ouar el ook i ngt oi ncr easey ourwebsi t econv er si onr at e, y ouwi l l hav et oconductABandmul t i v ar i at e t est i ng Thepr obl em i st hatmostt est i ngpr ogr amsst ar twi t houtanypl anni ngandt husf l wi t ht hefir st 6mont hs Wek nowt hi sfir sthandbecausewehav ebeenhel pi ngonl i necompani esconductt est i ngf orov er12 y ear s Ourgoal f r om t hi sgui det ohel py ouwi t hagoodi nt r oduct i ont omul t i v ar i at eandABt est i ng We wi l l cont i nueupdat i ngt hedi f f er entchapt er si nt hebookonr egul arbasi s, soi fy ouhav ecomment sor f eedback , pl easel et ’ sk now( @i nv esp) THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 01 C H A P TER WHATI SA/ BTESTI NG( SPLI TTESTI NG) ? WRI TTENBY:KHALI DSALEH C H A P TER WHATI SA/ BTESTI NG( SPLI TTESTI NG) ? I fv i si t or sar enotconv er t i ngony ourwebsi t e, t henobv i ousl y , t her ei sapr obl em t hati sst oppi ngt hem Youcangoaheadandasky ourdesi gnt eam t ocr eat enewdesi gns, butt hequest i onr emai ns: howdoy ouk nowt hatt henewdesi gnswi l l conv er tmor ev i si t or scompar edt ot heor i gi nal desi gn? Thati swher eABt est i ngcomesi nhandy A/ Bt est i ng( somet i mesr ef er r edt oasspl i tt est i ng)i st hepr ocessoft est i ngmul t i pl enewdesi gns ofawebpageagai nstt heor i gi nal desi gnoft hatpagewi t ht hegoal ofdet er mi ni ngwhi chdesi gngener at esmor econv er si ons Theor i gi nal desi gnofapagei susual l yr ef er r edt oast hecont r ol Thenewdesi gnsoft hepagear eusual l yr ef er r edt oast he“ v ar i at i ons” , “ chal l enger s”or“ r eci pes ”Thepr ocessoft est i ngwhi chpagedesi gngener at esmor econv er si onsi st y pi cal l yr ef er r edt oasa“ t est ”oran “ ex per i ment ” A“ conv er si on”wi l l v ar ybasedony ourwebsi t eandt hepagey ouar et est i ng Foranecommer cewebsi t e, aconv er si oncoul dbeav i si t or pl aci nganor der ForaSaaSwebsi t e, aconv er si oncoul dbeav i si t orsubscr i bi ngt ot heser v i ce Foral eadgener at i onwebsi t e, aconv er si on coul dbeav i si t orfil l i ngoutacont actf or m 1STEXAMPLE Thehomepageonanecommer cewebsi t er ecei v es100, 000v i si t or samont h Todet er mi nei ft her ei sawayt oi ncr ease conv er si ons, t hedesi gnt eam cr eat esonenewdesi gnf ort hehomepage ABt est i ngsof t war ei st henusedt or andoml yspl i tt hehomepagev i si t or sbet weent hecont r ol andt henewchal l enger So, 50, 000 v i si t or sar edi r ect edt ot hecont r ol and50, 000v i si t or sar edi r ect edt ot hechal l enger Si ncewear et est i ngwhi chdesi gngener at es mor eor der s( conv er si ons) , t henweuset heABt est i ngsof t war et ot r ackt henumberofconv er si onseachdesi gngener at es The A/ Bt est i ngsof t war ewi l l t hendet er mi net hewi nni ngdesi gnbasedont henumberofconv er si ons THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 03 2NDEXAMPLE Thehomepagef orabl ogr ecei v es3, 000v i si t or samont h Thepr i mar yconv er si ongoal f ort hehomepagei st ogetav i si t ort o subscr i bet ot heemai l l i stoft hebl og Thedesi gnercr eat esanewdesi gnf ort hebl oghomepagewhi chhi ghl i ght st he subscr i pt i onbox Thespl i tt est i ngsof t war ei susedt osend1, 500v i si t or st ot heor i gi nal pagedesi gn( cont r ol ) , andt het est i ngsof t war esends 1, 500v i si t or st ot henewdesi gn( chal l enger ) Thet est i ngsof t war et r ack st henumberofsubscr i ber s( conv er si ons)eachdesi gn gener at es A2015sur v eybyEconsul t ancyshowedt hat58%ofi t sr espondent sar econduct i ngA/ Bt est i ng: Buthowsuccessf ul i sABt est i ngi nhel pi ngcompani esi ncr easet hei rconv er si onr at es? A2017sur v eybyOpt i mi z el yshowst hatonl y25%ofal l A/ Bt est spr oducesi gni ficant l yposi t i v er esul t s Vi sual Websi t eOpt i mi z err epor t s t hatonl y12%ofal l A/ Bt estpr oducesi gni ficant l yposi t i v er esul t s Fi nal l y , dat af r om Googl eshowst hatonl y10%ofal l A/ Bt estpr oduce si gni ficant l yposi t i v er esul t s THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 04 CaseSt udy :WhatNet f l i xl ear nedf r om ABt est i ng? Backgr ound 46%ofsur v ey edNet f l i xv i si t or scompl nedt hatt hewebsi t edoesnotal l owt hem t ov i ewmov i et i t l esbef or esi gni ngupf ort heser v i ce So, Net f l i xdeci dedt or unanA/ Bt estont hei rr egi st r at i onpr ocesst oseei far edesi gnedr egi st r at i onpr ocesswi l l hel pi ncr easesubscr i pt i ons Cr eat i nganABt est Thenewdesi gndi spl ay edmov i et i t l est ov i si t or spr i ort or egi st r at i on TheNet f l i xt eam want edt ofindouti ft henewdesi gnwi t hmov i et i t l es woul dgener at emor er egi st r at i onscompar edt ot heor i gi nal desi gnwi t houtt het i t l es Thi swasanal y z edbyr unni nganA/ Bt estbet ween t henewdesi gnsagai nstt heor i gi nal desi gn Thet esthy pot hesi swasst r ght f or war d: Al l owi ngv i si t or st ov i ewav l abl emov i et i t l esbef or er egi st er i ngwi l l i ncr easet henumberofnew si gnups I nt hespl i tt est , t het eam i nt r oducedfiv edi f f er entchal l enger sagai nstt heor i gi nal desi gn Thet eam t henr ant het estt oseet hei mpact Whatwer et her esul t s? Resul t soft heABt estandanal ysi s Theor i gi nal desi gnconsi st ent l ybeatal l chal l enger s Ther eal anal y si shappensaf t ery ouconcl udeanA/ Bt est Whydi dt heor i gi nal desi gn beatal l newdesi gnsal t hough46%ofv i si t or ssai dt hatseei ngwhatt i t l esNet f l i xcar r i eswi l l per suadet hem t osi gnupf ort heser v i ce? Thet eam atNet f l i xgav et hr eedi f f er entr easonsofwhyt heor i gi nal desi gnbeatal l t hechal l enger s: 1.Net f l i xi sal laboutt heexper i ence:t hemor euser si nt er act edwi t ht hewebsi t e, t hemor et heyl ov et heex per i ence So, Net f l i xi smor e t hanj ustbr owsi ng 2.Si mpl i f ychoi ce:t heor i gi nal desi gn( t hecont r ol )showeduser soneopt i on: si gnupf ort heser v i ce Thenewdesi gnsof f er edv i si t or s mul t i pl eopt i ons( mul t i pl emov i es) Thi scompl i cat edt hechoi cewhi chv i si t or shadt omak e Mor echoi cesr esul t edi nf ewer conv er si ons 3.User sdonotal waysknowwhatt heywant :TheNet f l i xt eam ar guedt hatt estr esul t spoi ntt ot hef actt hatuser sdonotal way sk now whatt heywant THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 05 Whi l et hesemi ghtbev al i dex pl anat i ons, especi al l yt hesecondpoi nt , wewoul dar guet hatt her ei sanot herr easonal t oget her Coul di tbet hatv i si t or sfinal l yseeal l t hemov i eopt i onswhi chNet f l i xof f er sandt heydonotfindt hemov i esel ect i onconv i nci ng, sot hey deci det owal kaway ?I ft hati st hecase, i st hepr obl em wi t ht henewdesi gnsori si tapr obl em i nt hemov i esel ect i onwhi cht hesi t eof f er s? Howdoest heA/ Bt est i ngsof t war edet er mi net hewi nni ngdesi gn? Ati t scor e, ABt est i ngsof t war et r ack st henumberofv i si t or scomi ngt oeachdesi gni nanex per i mentandt henumberofconv er si onseach desi gngener at es Sophi st i cat edA/ Bt est i ngsof t war et r ack smuchmor edat af oreachv ar i at i on Asanex ampl e, Fi gPi i t r ack s: Conver si ons Pagevi ews Vi si t or s Bouncer at e Revenuepervi si t Exi t Revenue Sour ceoft r af fic Medi um oft r af fic Thespl i tt est i ngsof t war eusesdi f f er entst at i st i cal modest odet er mi neawi nneri nat est Thet wopopul armet hodsf ordet er mi ni nga wi nnerar eFr equent i standBay esi anmodel s Thespl i tt est i ngsof t war et r ack sconv er si onr at esf oreachdesi gn Howev er , decl ar i ngawi nneri naspl i tt estr equi r esmor et hangener at i ng asmal l i ncr easei nconv er si onr at escompar edt ot hecont r ol TheFr equent i stmodel usest womai nf act or st odet er mi net hewi nni ngdesi gn: Theconver si onr at ef oreachdesi gn:t hi snumberi sdet er mi nedbydi v i di ngt henumberofconv er si onsf oradesi gnbyt heuni que v i si t or sf ort hatdesi gn Theconfidencel evelf oreachdesi gn:ast at i st i cal t er mi ndi cat i ngt hecer t nt yt haty ourt estt hatwi l l pr oducet hesamer esul ti ft he sameex per i menti sconduct edacr ossmanysepar at edat aset si ndi f f er entex per i ment s Thi nkofconfidencel ev el ast hepr obabi l i t yofhav i ngar esul t So, i fachal l engerpr oducesa20%i ncr easei nconv er si onswi t ha95% confidence, t heny ouassumet haty ouhav eanex cel l entpr obabi l i t yofget t i ngt hesamer esul twhensel ect i ngt hatchal l engerasy our def aul tdesi gn I tal soi ndi cat est haty ouhav ea5%chancet haty ourt estr esul t swer eduet or andom chance, anda5%possi bi l i t yt haty ou f oundawr ongwi nner TheBay esi anmodel usest womai nf act or st odet er mi net hewi nni ngdesi gn: Theconver si onr at ef oreachdesi gn:asdefinedabov e Hi st or i calper f or mance:t hesuccessr at eofpr ev i ousl yr anA/ Bex per i ment sr anont hewebpage THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 06 “ “ Leoni dPek el i s, Opt i mi z el y ’ sfir sti nhousest at i st i ci an, ex pl nst hi sbysay i ng Bay esi anst at i st i cst akeamor ebot t omupappr oacht odat aanal ysi s.Thi s meanst hatpastknowl edgeofsi mi l arexper i ment si sencodedi nt oast at i st i cal devi ceknownasapr i or , andt hi spr i ori scombi nedwi t hcur r entexper i ment dat at omakeaconcl usi onont het estathand Wet y pi cal l yr el yonmul t i pl emet r i cswhendet er mi ni ngawi nni ngdesi gnf orat est Mostofourecommer cecl i ent suseacombi nat i onof conv er si onr at esandr ev enueperv i si tt odet er mi neafinal wi nneri nanex per i ment Sel ect i ngwhi chmet r i cswi l l dependony ourspeci ficsi t uat i on Howev er , i ti scr uci al t ochoosemet r i cst hathav eani mpactony ourbot t om l i ne Opt i mi z i ngf oral owerbounceorex i tr at eswi l l hav el i t t l edi r ectandmeasur abl edol l arv al uet omostbusi nesses Thet eam atBi ngwast r y i ngt ofindawayt oi ncr easet her ev enuewhi cht hesi t egener at esf r om ads Todoso, t heyi nt r oducedanew desi gnt hatemphasi z edhowsear chadsar edi spl ay ed.Thet eam t est edt henewdesi gnv s t heol ddesi gn Thespl i tt estr esul t sshoweda 30%i ncr easei nr ev enueperv i si t Thi s, howev er , wasduet oabugi nt hei rmai nsear chr esul t sal gor i t hm i nt henewdesi gn Thi sbugshowedv i si t or spoorsear chr esul t s And asar esul t , v i si t or swer ef r ust r at edandwer ecl i ck i ngonads Whi l et henewdesi gngener at edahi gherr ev enueperv i si t , t hi swasnotagoodl ongt er m st r at egy Thet eam deci dedt ost i ckt ot heol d desi gni nst ead Assi gni ngwei ght edt r af fict odi f f er entvar i at i ons MostABt est i ngsof t war eaut omat i cal l ydi v i desv i si t or sequal l ybet weendi f f er entv ar i at i ons Ther ear ehowev eri nst anceswher ey ouneedt oassi gndi f f er entwei ght st odi f f er entv ar i at i ons Forex ampl e, l et ’ st ak eanex per i mentt hathasanor i gi nal desi gnandt wochal l enger si ni t Thet est i ngt eam mi ghtwantt oassi gn50%of t hev i si t or st ot heor i gi nal desi gnandspl i tt her emai ni ng50%bet weenv ar i at i onsoneandt wo Shoul dyour unABt est i ngon100%ofyourvi si t or s? SomeConv er si onopt i mi z at i onex per t sdebat et hi squest i onatgr eatl engt hs Look i ngaty ouranal y t i cs, y oucant y pi cal l ynot i cet hatdi f f er entv i si t orsegment si nt er actdi f f er ent l ywi t hy ourwebsi t e Ret ur ni ngv i si t or s ( t hosewhov i si t edsi t epr ev i ousl y )gener al l yar emor eengagedwi t ht hewebsi t ecompar edt onewv i si t or s THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 07 Whenl aunchi nganewABt est , y ouwi l l not i cet hati nmanyi nst ances: Newv i si t or sr eacti nabet t erwaywi t hy ourex per i mentchal l enger s Ret ur ni ngv i si t or s, whoar eusedt oy ourcur r entdesi gn, r eactnegat i v el yt oy ournewdesi gns Thef actt hatnewv i si t or sconv er tathi gherr at eswi t hnewdesi gnscompar edt or et ur ni ngv i si t or si sat t r i but edt ot het heor yof moment um behav i or I fy ouy ourwebsi t eget sal ar genumberofv i si t or s, wer ecommendt haty oul aunchnewt est sf oronl ynewv i si t or sandobser v i nghowt hey r eactt oi t Af t ert hat , y oucanst ar tt het estf orr et ur ni ngv i si t or sandcompar et hei rr eact i onst ot henewdesi gnsi nt r oducedi nt he ex per i ment Hol dbackspl i tt est s 100% 50% 25% 5% 3 Unt i lwi nneri sdecl ar ed Wet y pi cal l yr ecommendr unni nghol dbackspl i tt est sf orl ar gerwebsi t est hatr ecei v et housandsofconv er si onspermont h I nt heset y pes oft est s, y oul auncht het est st oasmal l per cent ageofy oursi t ev i si t or s Forex ampl e, y oust ar twi t hl aunchi ngt het estt o10%ofy our v i si t or s I ft her esul t sar eencour agi ng, t heny ouex pandt het estt o25%, 50%, and100%ofy ourwebsi t ev i si t or s Ther ear esev er al adv ant agest or unni nghol dbackA/ Bt est s: Di scoveranyt est i ngbugs:Asy oul aunchanABt est , y ourdesi gnsmi ghthav ebugsi nt hem Byr unni ngt het estonasmal l per cent age ofy ourv i si t or s, onl yt hatt i nysegmentoft hev i si t or swi l l seet heer r or si nt henewdesi gns Thatwi l l gi v ey out heoppor t uni t yt ofixt hese bugsbef or er ol l i ngoutt het estt o100%ofy ourv i si t or s Reducer evenuer i sk:byr unni ngt het estonasmal l per cent ageofv i si t or s, y our educet her i skofhav i ngoneofy our st estv ar i at i on causi ngasi gni ficantdr opi nr ev enue I fy ouchooset or unhol dbackA/ Bt est s, mak esur et haty oust ar tanewt esteacht i mey ouchanget het r af fical l ocat i ongoi ngt hr ought he ex per i mentt oav oi danyst at i st i cal pr obl emswi t hr esul t s THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 08 C H A P TER 16MI STAKESTHATWI LLKI LLYOURA/ BTESTI NG ( ANDWHATYOUCANDOABOUTTHEM) Thescenehaschangedal otsi ncewefir stst ar t eddoi ngconv er si onopt i mi z at i onbacki n2006 Att hatt i me, manymar k et er sandCsui t eex ecut i v ewer enotconv i ncedt hatt her ei sal ott heycandoaboutt hei rwebsi t econv er si onr at e Thi ngsar emuchdi f f er entnowaday swi t hmanycompani esat t empt i ngt odepl oyABt est sont hei rwebsi t es Howev er , t hemaj or i t yofcompani escont i nuet opr i mar i l yi nv esti nv i si t or dr i v i ngact i v i t i esandsetl owerbudget sf orconv er t i ngv i si t or si nt o cust omer s I nv esp’ s2017st at eoft heconv er si onr at eopt i mi z at i onsur v eyshowedt hatmaj or i t yofcompani esar eal l ocat i ngl esst han8%oft hei r di gi t al mar k et i ngbudget st oconv er si onr at eopt i mi z at i on I fy ourcompanyi sst ar t i ngwi t hA/ Bormul t i v ar i at et est i ng, t hef ol l owi nggui dewi l l hel py ouav oi d16commonmi st ak eswehav eseen compani esf al l i nt odur i ngt hei rfir sty ear soft est i ng TESTI NGTHEWRONGPAGE I fy ouhav eawebsi t ewi t hal ar genumberofpages, al ar genumberofv i si t or sandal ar genumberofconv er si ons, t hent he possi bi l i t i esar eendl ess Foranecommer cewebsi t e, y oucanst ar ty ourconv er si onpr ogr am byf ocusi ngatt het opoft hef unnel pages, suchast he homepageorcat egor ypages Youcanal sost ar tatt hebot t om oft hef unnel pages, suchast hecar torcheck outpages I fy ouar eal eadgener at i onwebsi t e, y oucanst ar tbyopt i mi z i ngy ourhomepage, l andi ngpages, orcont actpages THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 61 Wheny ouchooset hewr ongpage, y oui nv estt i meandmoneyi napaget hatmi ghtnothav ear eal i mpactony ourbot t om l i ne Sur pr i si ngl y , manycompani escont i nuet opayl i t t l eat t ent i ont ot hei mpor t anceofsel ect i ngt her i ghtpaget oper f or m aspl i tt est on Wecov er edcr eat i ngapr i or i t i z edr esear choppor t uni t i esl i sti napr ev i ouschapt er Ourr esear choppor t uni t i esl i stf ordi f f er entCROpr oj ect scont nsonav er ageany wher ef r om 150t o250i t emsoni t Thi sl i sti s v er ypower f ul i ngui di ngy ourCROwor k Youwi l l nol ongerpi ckr andom pagesandr andom i t emst ot est Ev er yt esty oucr eat ei s pack edupbyr esear ch TESTI NGWI THOUTCREATI NGVI SI TORPERSONAS Test i nggi v esy ourv i si t or sav oi cei ny ourwebsi t edesi gnpr ocess I tv al i dat eswhatwor k sony oursi t eandwhatdoesnot But , bef or ey oust ar tt est i ng, y oumustunder st andy ourv i si t or satani nt i mat el ev el t ocr eat espl i tt est st hatappeal t ot hem Wedi scussedt hepr ocessofcr eat i ngper sonasi nsev er al ofourwebi nar sandwr ot eabouti ti nourbookConv er si onOpt i mi z at i on: TheAr tandSci enceofConv er t i ngPr ospect st oCust omer s Mostcompani eshav eadecentk nowl edgeoft hei rt ar getmar k et Thechal l engei showt oconv eyt hatk nowl edgei nt oact i onabl e mar k et i ngi nsi ght sony ourwebsi t e Per sonaspl ayacr uci al r ol ei nt hi spr ocess Let ’ ssayy ouar eanecommer cewebsi t et hatsel l sgi f tbask et sonl i ne Youhav ewor k edwi t hy ourmar k et i ngt eam t odefinet wo di f f er entsegment sofy ourt ar getmar k et : B2CSEGMENT B2BSEGMENT whi t ef emal es, age38t o48, col l egeeducat ed, wi t hannual i ncomeabov e$75, 000 Your av er ageor derv al uef ort hi ssegmenti s$125 cor por at ecl i ent s, wi t ht hepur chasedeci si onmade byanex ecut i v e Thesecompani esar egener at i ng bet ween10t o50mi l l i ondol l ar si nannual r ev enue Yourav er ageor derv al uef ort hi ssegmenti s$930 Theset wosegment spr ov i deat y pi cal f or matofmar k etsegment at i onandr set hecent r al quest i onofmar k et i ngdesi gn Howcany oudesi gny ourwebsi t et oappeal t ot heset wodi st i nct l ydi f f er entsegment s? Cr eat i ngper sonaswi l l hel py oui dent i f ywi t heachoft hesegment satmor edet l edl ev el Att heendoft heper sonacr eat i on pr ocessf ort hi swebsi t e, y oucoul dendupwi t hei ghtdi f f er entper sonas Let ’ st ak eoneoft hem asanex ampl e: Asy oucansee, Suz anr esembl esy ourt ar get mar k et Howev er , sher emov est heabst r actnat ur e oft hemar k et i ngdat a Asy oust ar tdesi gni ng di f f er entsect i onsoft hewebsi t e, y ouwi l l bet hi nk i ng ofSuz anandhowshewoul dr eactt ot hem Youmi ghtbet hi nk i ngt oy our sel f , “ t hi si sal l gr eat , buthowdoest hati mpactmyt est i ng? ” Pr operandsuccessf ul t est i ngusesper sonasi n cr eat i ngdesi gnv ar i at i onst hatchal l engey our ex i st i ngbasel i ne( cont r ol ) THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 62 Howwoul dyoucr eat eahomepaget estwhenyouar et hi nki ngofSuzan? Shei sacar i ngper sona, soy oucant estdi f f er entheadl i nest hatappeal t oher Shei sl ook i ngf oruni quegi f t s, soy oucant estdi f f er entdesi gnst hatemphasi z et heuni quenessoft hepr oduct sy ouof f er Pr i cei sanessent i al mot i v at i onf orSuz an, soy oucant estdi f f er entdesi gnst hatemphasi z epr i ci ng TESTI NGWI THOUTAHYPOTHESI S At est i nghy pot hesi si sapr edi ct i v est at ementaboutpossi bl epr obl emsonawebpage, andt hei mpactt hatfix i ngt hem mayhav e ony ourKPI I nourex per i ence, t her ear et wodi f f er entt y pesofi ssuescompani esr uni nt owheni tcomest ocr eat i nghy pot hesesf ordi f f er ent t est s: onas Let ’ st ak eoneoft hem asanex ampl e: 1.Manyt est er sdi smi sshy pot hesi sasal ux ur y So, t heycr eat eat estt hatgener at esr esul t s( posi t i v eornegat i v e) , butwheny ou askt hem aboutt her at i onal ebehi ndt het est , t heycannotex pl ni t 2.Ot her sunder st andt hei mpor t anceofhy pot hesi sbutt heycr eat emeani ngl essst at ement st hatdonothav ear eal i mpacton t est i ng Get t i ngdi sci pl i nedaboutcr eat i ngt esthy pot heseswi l l magni f yt hei mpactofy ourt estr esul t s Howdoyoucomeupwi t hat esthypot hesi si nt hefir stpl ace? Eachconv er si onpr obl emsony ourwebsi t eshoul di ncl udet hr eedi f f er entel ement si ni t : 1.Apr obl em st at ement 2.Howt hepr obl em wasi dent i fied 3.Ani ni t i al hy pot hesi sonhowt ofixt hepr obl em Asdi scussedpr ev i ousl y , y ouwi l l hav et osol i di f yy ouri ni t i al hy pot hesi sandcr eat eaconcr et ehy pot hesi s Ofcour se, y oucanal way si gnor et hepr ocess, t hr owt hi ngsatt hewal l , andpr ayt hatoneofy ournewdesi gnswi l l beatt hecont r ol Andy es, i tmi ghtwor ksomeoft het i me Buti twi l l notwor kmostoft het i me Andi twi l l i ndeednev erwor ki fy ouar el ook i ngf or r epeat abl eandsust nabl er esul t s NOTCONSI DERI NGMOBI LETRAFFI C Manywebsi t esar eget t i ngahi gherper cent ageoft hei rwebsi t et r af ficonmobi l edev i ces MostofourEur opeancl i ent sar e r epor t i ngany wher et hat50t o70%oft hei rt r af ficar r i v i ngatt hesi t eusi ngamobi l edev i ce Youcanonl yex pectt hesenumber st ogr owov ert henex tf ewy ear s So,whatshoul dyoudo? Eachconv er si onpr obl emsony ourwebsi t eshoul di ncl udet hr eedi f f er entel ement si ni t : 1.Det er mi net heper cent ageofy ourwebsi t et r af fict hatusesmobi l edev i cesf orbr owsi ng 2.Det er mi net het opt endev i cesv i si t or sar eusi ngt obr owsey ourwebsi t e 3.websi t eEv al uat et hebehav i orofmobi l et r af ficcompar edt odesk t opt r af ficf oral l oft hewebsi t ef unnel s Theset hr eest epsshoul dgi v ey oual otofact i onpoi nt st ot ak eony ourwebsi t e THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 63 Let ’ sseeanex ampl e: SOURCE Googl e Googl e Bi ng Facebook Emai l MEDI UM DEVI CE CATEGORY PRODUCT CARTTO TOPRODUCT TOCART CHECKOUT Or gani c Deskt op 40% 18% 40% Or gani c Mobi l e 43% 18% 22% Pai d Deskt op 31% 15% 33% Pai d Mobi l e 35% 13% 18% Pai d Deskt op 48% 22% 44% Pai d Mobi l e 52% 22% 25% Pai d Deskt op 32% 14% 37% Pai d Mobi l e 36% 13% 18% I nt er nal Deskt op 50% 35% 30% I nt er nal Mobi l e 55% 33% 18% ForaCROex per t , t hedat aabov epr ov i desaweal t hofi nf or mat i on Whenex ami ni ngt hef l owf r om cat egor yt opr oductpages, y ou wi l l not i cet hatBi ngpai dt r af ficout per f or msal l ot hert y pesofpai dt r af fic I tonl ycomessecondt ot heemai l campai gns Ont he ot herhand, Facebookpai dt r af ficunder per f or mscompar edt oot herchannel s I fy ouconsi dert hecat egor yt opr oductf l ow, mobi l et r af ficout per f or msdesk t opf oral l t r af ficsour cesandmedi ums Thi s i ndi cat est hatcat egor ypagedesi gnf ormobi l ei saccept abl et ouser s Howev er , not i cet hedr opi nmobi l eper f or mancewhen v i si t or sgett opr oductpages Thi ngsgetev enwor sewhenwest ar tev al uat i ngmobi l echeck out Thenumber sar et el l i ngust hatt hemobi l echeck outhasmuch hi gherabandonmentr at escompar edt odesk t op Ther eal i nv est i gat i onbegi nsher e Whyi si tt hatmobi l echeck outunder per f or ms? I si tapr obl em wi t ht hewebsi t edesi gnorusabi l i t y ?Ori si tt hecaset hatv i si t or suset hei rmobi l edev i cest obr owset hewebsi t e andt henwhent heyar er eadyt opl aceanor der , t heyuset hei rdesk t ops? MOBI LEVSDESKTOPEXPERI ENCE:THEAFTERMATHOFTESTI NG Whatbr i ngsv i si t or st oy ourwebsi t eonamobi l edev i cei sv ast l ydi f f er entf r om whatbr i ngst hem t oy ourwebsi t eonadesk t op Andt hati st her easony oushoul dr unsepar at emobi l eanddesk t opt est sondi f f er entpagesofy ourwebsi t e Thati sanobr ner But , howdoy ouhandl ei nst anceswher eawi nni ngdesi gnf oramobi l et estonapar t i cul arpagei sv er ydi f f er entf r om t hewi nni ng desi gnf oradesk t opex pi er ementf ort hatsamepage? I fy ouar el uck y , y ourt echni cal t eam i sabl et oser v et wocompl et el ydi f f er entex per i encest owebsi t ev i si t or sbasedont het y peof dev i cet heyar eusi ngt onav i gat e Hav i ngt wodi f f er entdesi gns( mobi l eanddesk t op)f ort hesamepagecancausemaj or dev el opmentpr obl ems Andasar esul t , wehav eseencompani esuseasi ngl el ay outf orbot hdesk t opandmobi l e Thatdoesnotsol v et hei ssue, i ti gnor esi t So, donotdot hat Di scussy ouropt i onswi t hy ourt echni cal t eam andfigur eoutaway t ohandl et hi si ssue THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 64 NOTRUNNI NGSEPARATETESTFORNEW VS.RETURNI NGVI SI TORS Ret ur ni ngv i si t or sar el oy al t oy oursi t e Theyar eusedt oi twi t hal l ofi t sconv er si onpr obl ems! Humansar ecr eat ur esofhabi t I nmanyi nst ances, wefindt hatr et ur ni ngv i si t or sconv er tatal owerr at ewhenwei nt r oducenew andbet t erdesi gns Thi shasbeenex pl nedi nt het heor yofmoment um behav i or Fort hi sr eason, weal way sr ecommendt est i ngnewwebsi t edesi gnswi t hnewv i si t or s Bef or ey out estnewdesi gns, y ouneedt oassesshowr et ur ni ngv i si t or si nt er actwi t hy ourwebsi t ecompar edt onewv i si t or s I fy ou r unGoogl eAnal y t i cs, y oucanv i ewv i si t orbehav i orbyaddi ngav i si t orsegmentt omostr epor t s Let ’ sex ami nehowv i si t or sv i ewdi f f er entpagesony ourwebsi t e Af t ery oul ogi nt oGoogl eAnal y t i cs, nav i gat et oBehav i or>Si t eCont ent>Al l Pages Googl ewi l l di spl ayt hepager epor tshowi ngdi f f er entmet r i csf ory ourwebsi t e Tov i ewhowr et ur ni ngv s newv i si t or si nt er actwi t hy ourwebsi t e, appl ysegment at i ont ot her epor t : THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 65 Sel ect“ NewUser s”and“ Ret ur ni ngUser s” Googl ewi l l nowdi spl ayt hesamer epor tsegment edbyt het y peofuser : Not i cet hedi f f er encef ort hi spar t i cul arwebsi t ei nt er msofbounceandex i tr at esf orr et ur ni ngv i si t or scompar edt onewv i si t or s Mostwebsi t esshowany wher ef r om 15%t o30%di f f er encei nmet r i csbet weent het wov i si t orsegment s I ft hedi f f er encei nmet r i csbet weennewandr et ur ni ngv i si t or si sl esst han10%, t heny oushoul dscr ut i ni z ey ourdesi gnt o under st andwhyr et ur ni ngv i si t or sr eactt hesamewayasnewv i si t or s Nex t , cr eat eat estspeci fical l yf ornewv i si t or s Mostspl i tt est i ngpl at f or mswi l l al l owy out osegmentv i si t or s I fy ourt est i ngsof t war edoesnotsuppor tt hi sf eat ur e, t henswi t cht osomet hi ngel se! I GNORI NGDI FFERENTTRAFFI CSOURCES Vi si t or sl andony ourwebsi t ef r om di v er set r af ficsour cesandmedi ums Youwi l l not i cet hatv i si t or sf r om di f f er entsour ces i nt er actwi t hy oursi t ei ndi f f er entway s Tr usti soneoft hefir standmostsi gni ficanti nf l uencest owhet herav i si t ori sper suadedt oconv er tony ourwebsi t eornot Oneof t hesubel ement soft r usti scont i nui t y THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 66 Cont i nui t ymeansy oumustmai nt naconsi st entmessagi nganddesi gnf r om t het r af ficsour ce/ medi um t ot hel andi ngpage Cr eat i ngt hesamet estf ordi f f er entt r af ficsour cesi gnor est hatv i si t or smi ghtseedi f f er entmessagi ngordesi gnsbef or el andi ng ony ourwebsi t e Toassesshowt r af ficsour cescani mpacty ourwebsi t e, f ol l owt heset hr eest eps: 1.Under st andhowdi f f er entt r af ficsour ces/ medi umsi nt er actwi t hy oursi t e 2.Anal y z er easonsf ordi f f er entv i si t ori nt er act i on( i fany ) 3.Cr eat esepar at et est sbasedont het r af ficsour ces/ medi ums Let ’ sseehowt hi si sdonei nGoogl eAnal y t i cs Fi r st , gener at et het r af ficsour ces/ medi umsr epor t Todot hi s, nav i gat et oAcqui si t i on>Al l Tr af fic>Sour ces/ Medi ums Googl ewi l l gener at et her epor tf ory ou Now, i ti sdi f ficul tt oassessal l y ourt r af fic sour ces Wer ecommendev al uat i ngei t hery ourt opt enort het r af ficsour cest hat dr i v emor et han50, 000v i si t or st oy ourwebsi t e Asy ouanal y z et her epor t , ex ami net hef ol l owi ngmet r i cs: Bouncer at e Pages/ Sessi on Exi tr at e Theconver si onr at ef ordi f f er entgoal s Askquest i onssuchas: Whi cht r af ficsour cesar edr i v i ngt hehi ghestconv er si ons? Doy ouseehi ghbouncer at esf orpai dt r af fic? Whi cht r af ficsour cesar edr i v i ngt hel owestbounceandex i tr at es? Thesest epshel pi ndet er mi ni ngi ft her ei sadi f f er encei nv i si t orbehav i orf or di f f er entt r af ficsour ces Nex t , y ouneedt odet er mi net hecausesofsuchdi f f er entbehav i or Thi swi l l r equi r eex ami ni ngeacht r af ficsour ce: Whatar ev i si t or sseei ngbef or el andi ngony ourwebsi t e? Doy ouhav econt r ol ov ert hedi spl ay / messagi ngt hatv i si t or sseebef or el andi ng ont hewebsi t e? Cany ouchangey ourl andi ngpaget omai nt ncont i nui t yi nt heex per i ence? TRYI NGTODOTOOMUCHI NONETEST Thi si soneoft hemi st ak eswef el l i ndur i ngt hefir sty earweconduct edA/ BandMVTt est i ng Ourcl i ent swant edt osee l ar gescal eex per i ment s Smal l t est sdi dnotconv i ncet hem Andi nst eadofex pl ni ngt ot hem whatwewer et r y i ngt oaccompl i sh, wecr eat edex t ensi v et est swher ewechangedt oomanyel ement satt hesamet i me Mostofourt est i ngpr oducedex cel l entr esul t s Asamat t eroff act , i n2007, 82%ofourt est i nggener at edanupl i f ti nconv er si onsand78%ofourt estpr oducedmor et han12% i ncr easei nconv er si onr at es THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 67 Thesear eamazi ngr esul t s.So,whatwast hepr obl em? Si ncewewer emak i ngt oomanychangesonapagef orev er yt est , wecoul dnoti sol at ewhatex act l ywasgener at i ngt heupl i f t So, ourt eam coul donl ymak eassumpt i ons Ev er yt estgener at edani ncr easei nconv er si onsduet osev ent oni nedi f f er entf act or s Thi sappr oachmi ghtbeok ayi fy ouar el ook i ngt odot wot ot hr eet est sandgetdonewi t ht est i ng( wedonotr ecommendi t ) Buti fy ouar el ook i ngf oral ongt er mt est i ngpr ogr am t hatt ak esacompanyf r om 2%t o9%conv er si onr at e, t hatappr oachwi l l undoubt edl yf l Ourappr oacht wel v ey ear sl at erl ook st r emendousl ydi f f er ent Ourt est i ngpr ogr amsar el asershar pf ocused Ev er yt estweper f or mr el i esonahy pot hesi s Thehy pot hesi scani nt r oducesev er al changesatt hesamet i met ot hepage, but r esear chback sal l changes Asi ngl ehy pot hesi saboutenhanci ngsoci al pr oofcoul dbei mpl ement edbydepl oy i nganewheadl i ne, newi mage, andanewl ay out Yes, t her ear esev er al el ement sbutt heyal l suppor tt heov er al l hy pot hesi s Let ’ st ak ear ecentev al uat i onwedi df orat opI RCE500r et l er Thei rpr oductpagessuf f er edf r om hi ghbouncer at es Vi si t or s wer ecl i ck i ngonPPCads, get t i ngt ot hepr oductpages, check i ngoutpr i cesf ordi f f er entpr oduct s, andt henl eav i ng Vi si t or swer e pr i ceshoppi ng Whatwer et her esul t sofr unni ngt heset est s? Thewebsi t ei ncr easedr ev enue( notconv er si ons! )byov er18% RUNNI NGA/ BTESTSWHENYOUARENOTREADY Ev er y onei st al k i ngaboutA/ Bandmul t i v ar i at et est i ng Thei deaofbei ngabl et oi ncr easey ourwebsi t er ev enuewi t houthav i ngt o dr i v emor ev i si t or st ot hewebsi t ei sf ant ast i c ButA/ Bt est i ngmi ghtnotwor kf orever ywebsi t e.Cer t nl y ,mul t i var i at et est i ngi snotf orever ysi t e Test i ngmi ghtnotwor kf ory oui nt woi nst ances: wheny oudonothav eenoughconv er si ons wheny oudonothav et hemi ndsetr equi r edt or unasuccessf ul CROpr ogr am 1.Youdonothaveenoughconver si ons I fy oudonothav eenought r af ficcomi ngt oy ourwebsi t eorenoughconv er si ons, t henspl i tt est i ngwi l l notwor kf ory ou Asmal l A/ Bt estt hathasonechal l engert oanor i gi nal desi gnr equi r est hatt hepar t i cul arpagey oupl ant ot estt ogener at ea mi ni mum of500conv er si onspermont h I ft hatpar t i cul arpagei sget t i ngl esst han500conv er si ons, t heny ourt est smi ghtr unf or t ool ongwi t houtconcl udi ng Wet y pi cal l ydonotst ar tA/ Bt est i ngwi t hacl i entunl esst hewebsi t ehas500conv er si onspermont h Ourst andar dpr ogr am r equi r est hatcl i ent scr osst he800conv er si onsperpl at f or m( desk t opormobi l e) Mul t i v ar i at et est i ngr equi r esmor econv er si onsandmor et r af ficcompar edt oABt est i ng Donotconsi derMVTt est i ngunl essy our websi t ehas10, 000conv er si onspermont h 2.Youdonothavet hemi ndsetneededt or unasuccessf ulconver si onpr ogr am Whi l et hefir str easonwhyaconv er si onpr ogr am f l si sst r ght f or war dt ofigur eout( y oul ookaty ourmont hl yconv er si ons) , t he secondpr obl em i smor edi f ficul tt oassess Thet r ut hi st hatnotev er yor gani z at i onorbusi nessi sr eadyf ort est i ng Test i ngr equi r esy out oadmi tt hatv i si t or smayhat ey ourex i st i ngwebsi t edesi gn Test i ngr equi r esy out oadmi tt hatsomedesi gnswhi chy ouhat ewi l l gener at emor econv er si ons/ sal esf ory ou THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 68 “ “ Test i ngr equi r essur r ender i ngt hefinal desi gndeci si ont oy ourv i si t or s Att hesur f ace, ev er ybusi nessownerort opex ecut i v ewi l lsayt hat t heyar ef ocusedont hei rr ev enue.Butaf t err unni ngov er500+ conv er si onopt i mi zat i onpr oj ect swi t hov er10, 000+spi tt est si n t hem, wecansi mpl yst at et hatt hi si snott hecase Wehav eseenbusi nessowner sr ej ectt her esul t soft est i ngt hatgener at ed32%upl i f ti nconv er si onswi t h99%confidencebecause t heyl i k edt heor i gi nal desi gnandcoul dnotbeart hei deaofchangi ngi t Wehav eseenex ecut i v esr ej ectt her esul t soft est i ngt hatgener at ed25%upl i f ti nconv er si onswi t h95%confidencebecauset hey hat edt hewi nni ngdesi gn Wehav eal soseent est i ngpr ogr amsf l because, whi l et heCEOoft hecompanywascommi t t edt ot est i ng, t het eam wasnotsol d ont hei dea Formanycompani es, spl i tt est i ngr equi r esacompl et ecul t ur eshi f t Tomak esur ei t sr esul t swi l l hav eadi r ectandsi gni ficant i mpactony ourbot t om l i ne, ev er y one–andwedomeanev er y one–mustbecompl et el ycommi t t edt oi t CALLI NGTHETESTTOOSOON Your unt het estand, af ewday sl at er , y ourt est i ngsof t war edecl ar esawi nni ngdesi gn Ev er y onei sex ci t edaboutt heupl i f t You st opt het estandmak et hewi nni ngchal l engery ourdef aul tdesi gn Youex pectt oseey ourconv er si onr at ei ncr ease Buti tdoesnot Why ? Becauset het estwasconcl udedt ooqui ck l y Mostt est i ngsof t war edecl ar eswi nner saf t erachi ev i nga95%confidencel ev el r egar dl essoft henumberofv i si t or sandconv er si onsr epor t edf ordi f f er entv ar i at i ons Manyspl i tt est i ngpl at f or msdonott ak ei nt oaccountt henumberofconv er si onst heor i gi nal desi gnandt hev ar i at i onsr ecor ded I f t het esti sal l owedt or unl ongenough, y ouwi l l not i cet hatt heobser v edupl i f tsl owl ydi sappear s So,howdoyoudealwi t ht hi s? 1.Regar dl essoft heset upi nt hespl i tt est i ngpl at f or m, mak esur ey ouadj usti tt or equi r eami ni mum of500conv er si onsf ort he or i gi nal desi gnandt hewi nni ngchal l enger 2.Runy ourt est sf orami ni mum ofsev enday ssot haty ourt estwi l l r unonev er ydayoft heweekt oaccountf orf l uct uat i ons mi ghthappenbet weendi f f er entday s( Wet y pi cal l yr unt est sf orami ni mum oft woweek sdependi ngont hebusi nesscy cl ef or t hepar t i cul arbusi ness) THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 69 1 CALLI NGTHETESTTOOLATE Youhav enocont r ol ov erex t er nal f act or swheny our unaspl i tt est Thesef act or scanpol l ut et her esul t sofy ourt est i ngpr ogr am Ther ear et hr eedi f f er entcat egor i esofex t er nal f act or st hatcoul di mpacty ourex per i ment s: Gener al mar k ett r ends: asuddendownt ur ni nt heeconomy , f orex ampl e Compet i t i v ef act or s: acompet i t orr unni ngamassi v emar k et i ngcampai gn Tr af ficf act or s: changei nt hequal i t yofor gani corpai dt r af fic Al l oft hesef act or scoul di mpactt her esul t sofy ourt est i ngatnof aul toft het est i ngpr ogr am i t sel f Fort hi sr eason, wehi ghl y r ecommendl i mi t i ngt het i mespanofanyspl i tt estf ornol ongert han30day s Wehav eseencompani esr equi r et est st or unf ort wot ot hr eemont hst r y i ngt oachi ev econfidenceonat est I nt hepr ocess, t hey pol l ut et hei rt est i ngdat a I nfix edhor i z ont est i ng, y oushoul dcal cul at et henumberofv i si t or sr equi r edt ogot hr ought het estbef or el aunchi ngany ex per i ment I nt hi sappr oach, y ouwi l l usey ourcur r entconv er si onr at e, si gni ficancel ev el , andt heex pect edupl i f tt odet er mi ne r equi r edsampl esi z e Afinal not eonsi gni ficancel ev el s: Remembert hatachi ev i ng95%si gni ficancei snotagoal seti nst one Si gni ficancel ev el spr ov i de agener al t r endl i net hatt het estr esul t sar eposi t i v eandconsi st ent WHENTECHNOLOGYBECOMESAPROBLEM Thepur poseoft est i ngi st oi ncr easey ourconv er si onr at esandy ourr ev enue Dev el oper ssomet i messt r uggl et of ocusont hi s f act , especi al l ywhent heygetf asci nat edbyapar t i cul arpi eceofsof t war et hatcompl i cat est esti mpl ement at i on Asagoal , mostspl i tt est sshoul dnott ak el ongert hant hr eet ofiv eday st oi mpl ement Smal l ert est s, wi t hf ewchangest oapage, mi ghtt ak eaf ewhour st oi mpl ement Youmustk eepi nmi ndt hatt hefir stt wospl i tt est sy oudepl oyony ourwebsi t ewi l l t ak eal i t t l el ongert oi mpl ementasy our dev el opmentt eam get susedt ot hespl i tt est i ngpl at f or my ousel ect ed Howev er , i fy ounot i cet hat , ov erasi x mont hper i od, al l ofy ourt est sar et ak i ngov eraweekt ot enday st oi mpl ement , t heny ou MUSTassesst hecausef ort hedel ay : I st het est i ngpl at f or mt oocompl i cat edandanov er k i l l f ort het y peoft est i ngy ouar edoi ng? I sy ourwebsi t eorappl i cat i oncodenotdev el opedwi t hpr operst andar dswhi char ecausi ngt hedel ay s? Doesy ourdev el opmentt eam hav eagoodhandl eoni mpl ement i ngt est sorar et heyst r uggl i ngwi t hev er yt est ? Ar ey oucr eat i nganddepl oy i ngcompl i cat edt est st hatr equi r esi gni ficantt i met odev el op? Gol denr ul e: At est i ngpr ogr am shoul ddepl oyt wot osi xt est spermont h RUNNI NGSUCCESSFULSI MUL TANEOUSSPLI TTESTS Launchi ngsi mul t aneousspl i tt est sex pedi t esy ourt est i ngpr ogr am I nst eadofwai t i ngf oreacht estt oconcl ude, y our unsev er al t est satt hesamet i me I fy ouar er unni ngsi mul t aneousspl i tt est si nsepar at eswi ml anes, t hent hi swi l l notcausey ouani ssue Howev er , i ft hesamev i si t or sar enav i gat i ngt hr oughy ourwebsi t eandseei ngdi f f er entt est s, y oucoul dbecanni bal i z i ngy our t est i ngdat a THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 70 I magi net hescenar i oofr unni ngat estont hehomepage, Att hesamet i me, y our unat estont hepr oductpages, wi t ht hr eechal l enger st ot heor i gi nal desi gn Av i si t or wi t ht wochal l enger st ot heor i gi nal desi gn Av i si t ormi ght mi ghtv i ewanyoft hef ol l owi ngconfigur at i ons: v i ewanyoft hef ol l owi ngconfigur at i ons: H0 Or i gi nal homepagedesi gn P0 Or i gi nal pr oductdesi gn H1 Chal l enger1 P1 Chal l enger1 H2 Chal l enger2 P2 Chal l enger2 H3 Chal l enger3 I nt hi sscenar i o, ast hev i si t ornav i gat esf r om t hehomepaget ot hepr oductpages, hecanseeanyoft hef ol l owi ngcombi nat i ons ofdesi gns: H0, P0 Or i gi nal homepage, or i gi nal pr oductpage H0, P1 Or i gi nal homepage, pr oductpagechal l enger1 H0, P2 Or i gi nal homepage, pr oductpagechal l enger2 H1, P0 Homepagechal l enger1, or i gi nal pr oductpage H1, P1 Homepagechal l enger1, pr oductpagechal l enger1 H1, P2 Homepagechal l enger1, pr oductpagechal l enger2 H2, P0 Homepagechal l enger2, or i gi nal pr oductpage H2, P1 Homepagechal l enger2, pr oductpagechal l enger1 H2, P2 Homepagechal l enger2, pr oductpagechal l enger2 10 H3, P0 Homepagechal l enger3, or i gi nal pr oductpage 11 H3, P1 Homepagechal l enger3, pr oductpagechal l enger1 12 H3, P2 Homepagechal l enger3, pr oductpagechal l enger2 Yourt wosepar at eandst r ght f or war dt est sendupi mpact i ngeachot her So, doest hatmeant haty oushoul dnev err unsi mul t aneousspl i tt est s? No, wer unspl i tt est ssi mul t aneousl yonpr oj ect swi t ht hef ol l owi nggui del i nes: 1.Wet ar getdi f f er entt r af ficanddev i cet y pest ospl i tt het r af fic 2.Af t erwedecl ar eawi nnerf orat est , wer unt hewi nneragai nstt hecont r ol i naheadt oheadmat chi naper i odwher enoot her t est sar er unni ng THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 71 MI SSI NGTHEI NSI GHTS Ther eal i mpactofconv er si onopt i mi z at i ont ak espl aceaf t ery ouconcl udeeachofy ourspl i tt est s Thi si mpacti sbynomeans l i mi t edt oani ncr easei nconv er si onr at es Yes, seei ngani ncr easei nconv er si onr at esi sf ant ast i c! Butt her ei sasecr ett oampl i f y i ngt her esul t sofanyt estbydepl oy i ngt hemar k et i ngl essonsy oul ear nedacr ossdi f f er entchannel s andv er t i cal s Let ’ sputt hi si nper spect i v e Wewor k edwi t ht hel ar gestsat el l i t epr ov i deri nNor t hAmer i cahel pi ngt hem t est sdi f f er entl andi ngpagesf ort hei rPPCcampai gn Thet est i ngpr ogr am wasv er ysuccessf ul , gener at i ngsi gni ficanti ncr easesi nconv er si onr at es Aswei mpl ement edt het est i ngpr ogr am, t hei rdi gi t al mar k et i ngdi r ect orask edi fwecoul duset hesamel essonswel ear nedf r om spl i tt est i ngi nt hei rnewspaperadv er t i si ng Thi swasanewchal l enge Woul dof f l i nebuy er sr eactt hesamewayt oadv er t i si ngasonl i nebuy er s? Ther ewasonl yonewayt ofindout Wehadt ot esti t Eacht estwasbui l tusi ngahy pot hesi s Weappl i edt hathy pot hesi st obot honl i net est saswel l ast henewspaperadv er t i si ng We r ant hr eedi f f er entt est sbot honl i neandof f l i ne Eacht estgener at edanupl i f ti nconv er si onsi nbot honl i neandof f l i ne Butt hi ngsdi dnotst opt her e Wet henappl i edt hesamel essonst omai l er swhi cht hecompanysentt or esi dent i al addr esses Agai n, wesawupl i f t si nconv er si ons I fy ouf ol l owaconv er si onopt i mi z at i onmet hodol ogy , t heny ouwi l l beabl et ot ak el essonsf r om y ourt est i ngandappl yt hem agai n andagai n YOURFOLLOW UPTESTS Notev er yt estwi l l gener at eani ncr easei nconv er si ons Asy ouconcl udey ourex per i ment s, y ouwi l l hav et odeci deont henex t st epsf oreacht est Ihav eseencl i ent sandt eamsgetf r ust r at edbecauset heyar enotabl et oi ncr easeconv er si onsonapar t i cul ar page So, t heymov eont oot hersect i onsont hewebsi t e Thepost t estanal y si sshoul ddet er mi neoneoff ourpossi bl enex tst eps: 1.I t er at eont het est :conductf ur t hert est sont hepage, finet uni ngt het estdesi gnort r y i ngdi f f er enti mpl ement at i ont ot he or i gi nal hy pot hesi s 2.Testnewr esear choppor t uni t i es:t estr esul t spoi ntt oanewhy pot hesi st hatshoul dbet est edont hepage 3.I nvest i gat ef ur t her :t estr esul t sshowt haty ouneedt odi gf ordeeperi nsi ght sbef or ey oudet er mi nenex tst eps 4.Pi vot :Yourt est i ngdat aandanal y si scl ear l yi ndi cat et haty ouhadt hewr onghy pot hesi st ost ar twi t h Ti met ol ookf orot her i ssuesont hepage THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 72 NOTDOCUMENTI NGEVERYTHI NG Aconv er si onopt i mi z at i onpr ogr am i sdocument at i oni nt ensi v e Youshoul dr ecor dev er yl i t t l edet l Youmustdocument : Yourqual i t at i v eanal y si sr esear chandfindi ngs Yourquant i t at i v eanal y si sandconcl usi ons Ev er ypageanal y si sy ouconduct Ev er yhy pot hesi sy oumak e Scr eencapt ur esofev er ydesi gny oudepl oy Test i ngdat a Post t estanal y si s Manycompani esdonotpaycl oseat t ent i ont ot hei mpor t anceofdocument at i on Theydi scov eri t si mpor t ancewhent heycome backt ot het est i ngr esul t saf ewmont hsl at er Whent her ei snodocument at i onorwheni ti sscat t er edi nacr ossmul t i pl eemai l s, t heyst r uggl et or ememberwhyt heymadeapar t i cul archange Buti ti st ool at e Fr om t hev er yst ar tofanyCROpr oj ect , deci deont hemet hody ouwi l l uset odocumentev er y t hi ngandusei tt hor oughl y THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 73 I NCREDI BLEREVENUE& PROFI TSGROWTH DATAI NFORMEDANDCUSTOMEROBSESSEDI NSI GHTS TODRI VEPROFI TSFORYOURBUSI NESS OURSERVI CES Endt oEndDi gi t al Opt i mi z at i on St r at egi c Di scov er y Landi ngPage Opt i mi z at i on Desi gnf or Gr owt h Conv er si onRat eOpt i mi z at i on Pr of essi onal Ser v i ces Conv er si onRat e Audi t&St r at egy CRO Tr ni ng LEARN MORE © 2018Al lr i ght sr eser v ed.I nv esp www i nv espcr o com © 2018Al lr i ght sr eser v ed.I nv esp ... f or m anyoft hei rcount er par t si nt hi sar ea THEDEFI NI TI VEGUI DETO MUL TI VARI ATEANDABTESTI NG 25 WHATI STHECOSTOFRUNNI NGTHEABTESTI NGSOFTWARE? Thecostofr unni ngt est i ngsof t war... ementx…xNumberofv ar i at i onsoft heNt hel ement Thenumberofpagev ar i at i onscangr owv er yf ast Somet est i ngsof t war eal l owsy out ot ensoft housands( somet i mesmi l l i ons )of v ar i at i onsofasi... yi npr i ceandr angeany wher ef r om $50/ mont ht ot ensoft housandsofdol l ar s FREEABTESTI NGTOOLS Theset ool spr ov i det hecust omert heabi l i t yt oconductspl i tt est i ngf r eeofchar ge,