CHAPTER 16 Principal Component Analysis: The Olympic Heptathlon 16.1 Introduction The pentathlon for women was first held in Germany in 1928 Initially this consisted of the shot put, long jump, 100m, high jump and javelin events held over two days In the 1964 Olympic Games the pentathlon became the first combined Olympic event for women, consisting now of the 80m hurdles, shot, high jump, long jump and 200m In 1977 the 200m was replaced by the 800m and from 1981 the IAAF brought in the seven-event heptathlon in place of the pentathlon, with day one containing the events 100m hurdles, shot, high jump, 200m and day two, the long jump, javelin and 800m A scoring system is used to assign points to the results from each event and the winner is the woman who accumulates the most points over the two days The event made its first Olympic appearance in 1984 In the 1988 Olympics held in Seoul, the heptathlon was won by one of the stars of women’s athletics in the USA, Jackie Joyner-Kersee The results for all 25 competitors in all seven disciplines are given in Table 16.1 (from Hand et al., 1994) We shall analyse these data using principal component analysis with a view to exploring the structure of the data and assessing how the derived principal component scores (see later) relate to the scores assigned by the official scoring system 16.2 Principal Component Analysis The basic aim of principal component analysis is to describe variation in a set of correlated variables, x1 , x2 , , xq , in terms of a new set of uncorrelated variables, y1 , y2 , , yq , each of which is a linear combination of the x variables The new variables are derived in decreasing order of ‘importance’ in the sense that y1 accounts for as much of the variation in the original data amongst all linear combinations of x1 , x2 , , xq Then y2 is chosen to account for as much as possible of the remaining variation, subject to being uncorrelated with y1 – and so on, i.e., forming an orthogonal coordinate system The new variables defined by this process, y1 , y2 , , yq , are the principal components The general hope of principal component analysis is that the first few components will account for a substantial proportion of the variation in the original variables, x1 , x2 , , xq , and can, consequently, be used to provide a conve285 © 2010 by Taylor and Francis Group, LLC Joyner-Kersee (USA) John (GDR) Behmer (GDR) Sablovskaite (URS) Choubenkova (URS) Schulz (GDR) Fleming (AUS) Greiner (USA) Lajbnerova (CZE) Bouraga (URS) Wijnsma (HOL) Dimitrova (BUL) Scheider (SWI) Braun (FRG) Ruotsalainen (FIN) Yuping (CHN) Hagger (GB) Brown (USA) Mulliner (GB) Hautenauve (BEL) Kytola (FIN) Geremias (BRA) Hui-Ing (TAI) Jeong-Mi (KOR) Launa (PNG) heptathlon data Results Olympic heptathlon, Seoul, 1988 hurdles 12.69 12.85 13.20 13.61 13.51 13.75 13.38 13.55 13.63 13.25 13.75 13.24 13.85 13.71 13.79 13.93 13.47 14.07 14.39 14.04 14.31 14.23 14.85 14.53 16.42 © 2010 by Taylor and Francis Group, LLC highjump 1.86 1.80 1.83 1.80 1.74 1.83 1.80 1.80 1.83 1.77 1.86 1.80 1.86 1.83 1.80 1.86 1.80 1.83 1.71 1.77 1.77 1.71 1.68 1.71 1.50 shot 15.80 16.23 14.20 15.23 14.76 13.50 12.88 14.13 14.28 12.62 13.01 12.88 11.58 13.16 12.32 14.21 12.75 12.69 12.68 11.81 11.66 12.95 10.00 10.83 11.78 run200m 22.56 23.65 23.10 23.92 23.93 24.65 23.59 24.48 24.86 23.59 25.03 23.59 24.87 24.78 24.61 25.00 25.47 24.83 24.92 25.61 25.69 25.50 25.23 26.61 26.16 longjump 7.27 6.71 6.68 6.25 6.32 6.33 6.37 6.47 6.11 6.28 6.34 6.37 6.05 6.12 6.08 6.40 6.34 6.13 6.10 5.99 5.75 5.50 5.47 5.50 4.88 javelin 45.66 42.56 44.54 42.78 47.46 42.82 40.28 38.00 42.20 39.06 37.86 40.28 47.50 44.58 45.44 38.60 35.76 44.34 37.76 35.68 39.48 39.64 39.14 39.26 46.38 run800m 128.51 126.12 124.20 132.24 127.90 125.79 132.54 133.65 136.05 134.74 131.49 132.54 134.93 142.82 137.06 146.67 138.48 146.43 138.02 133.90 133.35 144.02 137.30 139.17 163.43 score 7291 6897 6858 6540 6540 6411 6351 6297 6252 6252 6205 6171 6137 6109 6101 6087 5975 5972 5746 5734 5686 5508 5290 5289 4566 PRINCIPAL COMPONENT ANALYSIS Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 286 Table 16.1: Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 PRINCIPAL COMPONENT ANALYSIS 287 nient lower-dimensional summary of these variables that might prove useful for a variety of reasons In some applications, the principal components may be an end in themselves and might be amenable to interpretation in a similar fashion as the factors in an exploratory factor analysis (see Everitt and Dunn, 2001) More often they are obtained for use as a means of constructing a low-dimensional informative graphical representation of the data, or as input to some other analysis The low-dimensional representation produced by principal component analysis is such that n n d2rs − dˆ2rs r=1 s=1 is minimised with respect to dˆ2rs In this expression, drs is the Euclidean distance (see Chapter 17) between observations r and s in the original q dimensional space, and dˆrs is the corresponding distance in the space of the first m components As stated previously, the first principal component of the observations is that linear combination of the original variables whose sample variance is greatest amongst all possible such linear combinations The second principal component is defined as that linear combination of the original variables that accounts for a maximal proportion of the remaining variance subject to being uncorrelated with the first principal component Subsequent components are defined similarly The question now arises as to how the coefficients specifying the linear combinations of the original variables defining each component are found? The algebra of sample principal components is summarised briefly The first principal component of the observations, y1 , is the linear combination y1 = a11 x1 + a12 x2 + , a1q xq whose sample variance is greatest among all such linear combinations Since the variance of y1 could be increased without limit simply by increasing the coefficients a⊤ = (a11 , a12 , , a1q ) (here written in form of a vector for convenience), a restriction must be placed on these coefficients As we shall see later, a sensible constraint is to require that the sum of squares of the coefficients, a⊤ a1 , should take the value one, although other constraints are possible The second principal component y2 = a⊤ x with x = (x1 , , xq ) is the linear combination with greatest variance subject to the two conditions a⊤ a2 = and a⊤ a1 = The second condition ensures that y1 and y2 are uncorrelated Similarly, the jth principal component is that linear combination yj = a⊤ j x which has the greatest variance subject to the conditions a⊤ a = and j j a⊤ j = for (i < j) To find the coefficients defining the first principal component we need to choose the elements of the vector a1 so as to maximise the variance of y1 subject to the constraint a⊤ a1 = © 2010 by Taylor and Francis Group, LLC Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 288 PRINCIPAL COMPONENT ANALYSIS To maximise a function of several variables subject to one or more constraints, the method of Lagrange multipliers is used In this case this leads to the solution that a1 is the eigenvector of the sample covariance matrix, S, corresponding to its largest eigenvalue – full details are given in Morrison (2005) The other components are derived in similar fashion, with aj being the eigenvector of S associated with its jth largest eigenvalue If the eigenvalues of S are λ1 , λ2 , , λq , then since a⊤ j aj = 1, the variance of the jth component is given by λj The total variance of the q principal components will equal the total variance of the original variables so that q λj = s21 + s22 + · · · + s2q j=1 where s2j is the sample variance of xj We can write this more concisely as q λj = trace(S) j=1 Consequently, the jth principal component accounts for a proportion Pj of the total variation of the original data, where Pj = λj trace(S) The first m principal components, where m < q, account for a proportion m λj P (m) = j=1 trace(S) When the variables are on very different scales principal component analysis is usally carried out on the correlation matrix rather than the covariance matrix 16.3 Analysis Using R To begin it will help to score all seven events in the same direction, so that ‘large’ values are ‘good’ We will recode the running events to achieve this; R> data("heptathlon", package = "HSAUR2") R> heptathlon$hurdles heptathlon$run200m heptathlon$run800m score plot(heptathlon[,-score]) 1.50 1.75 36 289 42 1.75 16 1.50 highjump 10 13 shot 6.5 run200m 42 5.0 longjump javelin run800m 20 40 36 Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 hurdles Figure 16.1 10 13 16 5.0 6.5 20 40 Scatterplot matrix for the heptathlon data (all countries) is a positive relationship between the results for each pairs of events The exception are the plots involving the javelin event which give little evidence of any relationship between the result for this event and the results from the other six events; we will suggest possible reasons for this below, but first we will examine the numerical values of the between pairs events correlations by applying the cor function R> round(cor(heptathlon[,-score]), 2) hurdles highjump shot run200m hurdles highjump shot run200m longjump javelin run800m 1.00 0.81 0.65 0.77 0.91 0.01 0.78 0.81 1.00 0.44 0.49 0.78 0.00 0.59 0.65 0.44 1.00 0.68 0.74 0.27 0.42 0.77 0.49 0.68 1.00 0.82 0.33 0.62 © 2010 by Taylor and Francis Group, LLC 290 Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 longjump javelin run800m PRINCIPAL COMPONENT ANALYSIS 0.91 0.01 0.78 0.78 0.74 0.00 0.27 0.59 0.42 0.82 0.33 0.62 1.00 0.07 0.70 0.07 1.00 -0.02 0.70 -0.02 1.00 Examination of these numerical values confirms that most pairs of events are positively correlated, some moderately (for example, high jump and shot) and others relatively highly (for example, high jump and hurdles) And we see that the correlations involving the javelin event are all close to zero One possible explanation for the latter finding is perhaps that training for the other six events does not help much in the javelin because it is essentially a ‘technical’ event An alternative explanation is found if we examine the scatterplot matrix in Figure 16.1 a little more closely It is very clear in this diagram that for all events except the javelin there is an outlier, the competitor from Papua New Guinea (PNG), who is much poorer than the other athletes at these six events and who finished last in the competition in terms of points scored But surprisingly in the scatterplots involving the javelin it is this competitor who again stands out but because she has the third highest value for the event It might be sensible to look again at both the correlation matrix and the scatterplot matrix after removing the competitor from PNG; the relevant R code is R> heptathlon round(cor(heptathlon[,-score]), 2) hurdles highjump shot run200m longjump javelin run800m hurdles highjump shot run200m longjump javelin run800m 1.00 0.58 0.77 0.83 0.89 0.33 0.56 0.58 1.00 0.46 0.39 0.66 0.35 0.15 0.77 0.46 1.00 0.67 0.78 0.34 0.41 0.83 0.39 0.67 1.00 0.81 0.47 0.57 0.89 0.66 0.78 0.81 1.00 0.29 0.52 0.33 0.35 0.34 0.47 0.29 1.00 0.26 0.56 0.15 0.41 0.57 0.52 0.26 1.00 The correlations change quite substantially and the new scatterplot matrix in Figure 16.2 does not point us to any further extreme observations In the remainder of this chapter we analyse the heptathlon data with the observations of the competitor from Papua New Guinea removed Because the results for the seven heptathlon events are on different scales we shall extract the principal components from the correlation matrix A principal component analysis of the data can be applied using the prcomp function with the scale argument set to TRUE to ensure the analysis is carried out on the correlation matrix The result is a list containing the coefficients defining each component (sometimes referred to as loadings), the principal component scores, etc The required code is (omitting the score variable) R> heptathlon_pca print(heptathlon_pca) © 2010 by Taylor and Francis Group, LLC ANALYSIS USING R R> score plot(heptathlon[,-score]) 1.85 36 42 3.0 1.70 291 1.5 1.85 16 1.70 highjump 10 13 shot 6.5 run200m 42 5.5 longjump javelin 20 run800m 30 40 36 Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 hurdles 1.5 3.0 Figure 16.2 10 13 16 5.5 6.5 20 30 40 Scatterplot matrix for the heptathlon data after removing observations of the PNG competitor Standard deviations: [1] 2.0793 0.9482 0.9109 0.6832 0.5462 0.3375 0.2620 Rotation: hurdles highjump shot run200m longjump javelin run800m PC1 -0.4504 -0.3145 -0.4025 -0.4271 -0.4510 -0.2423 -0.3029 PC2 0.05772 -0.65133 -0.02202 0.18503 -0.02492 -0.32572 0.65651 © 2010 by Taylor and Francis Group, LLC PC3 PC4 PC5 PC6 -0.1739 0.04841 -0.19889 0.84665 -0.2088 -0.55695 0.07076 -0.09008 -0.1535 0.54827 0.67166 -0.09886 0.1301 0.23096 -0.61782 -0.33279 -0.2698 -0.01468 -0.12152 -0.38294 0.8807 0.06025 0.07874 0.07193 0.1930 -0.57418 0.31880 -0.05218 Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 292 PRINCIPAL COMPONENT ANALYSIS PC7 hurdles -0.06962 highjump 0.33156 shot 0.22904 run200m 0.46972 longjump -0.74941 javelin -0.21108 run800m 0.07719 The summary method can be used for further inspection of the details: R> summary(heptathlon_pca) Importance of components: PC1 Standard deviation 2.1 Proportion of Variance 0.6 Cumulative Proportion 0.6 PC2 0.9 0.1 0.7 PC3 PC4 PC5 PC6 PC7 0.9 0.68 0.55 0.34 0.26 0.1 0.07 0.04 0.02 0.01 0.9 0.93 0.97 0.99 1.00 The linear combination for the first principal component is R> a1 a1 hurdles highjump shot run200m longjump -0.4503876 -0.3145115 -0.4024884 -0.4270860 -0.4509639 javelin run800m -0.2423079 -0.3029068 We see that the 200m and long jump competitions receive the highest weight but the javelin result is less important For computing the first principal component, the data need to be rescaled appropriately The center and the scaling used by prcomp internally can be extracted from the heptathlon_pca via R> center scale hm drop(scale(hm, center = center, scale = scale) %*% + heptathlon_pca$rotation[,1]) Joyner-Kersee (USA) -4.757530189 Sablovskaite (URS) -1.288135516 Fleming (AUS) -0.953445060 Bouraga (URS) -0.522322004 Scheider (SWI) 0.003014986 © 2010 by Taylor and Francis Group, LLC John (GDR) -3.147943402 Choubenkova (URS) -1.503450994 Greiner (USA) -0.633239267 Wijnsma (HOL) -0.217701500 Braun (FRG) 0.109183759 Behmer (GDR) -2.926184760 Schulz (GDR) -0.958467101 Lajbnerova (CZE) -0.381571974 Dimitrova (BUL) -1.075984276 Ruotsalainen (FIN) 0.208868056 ANALYSIS USING R Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 Yuping (CHN) 0.232507119 Mulliner (GB) 1.880932819 Geremias (BRA) 2.770706272 293 Hagger (GB) 0.659520046 Hautenauve (BEL) 1.828170404 Hui-Ing (TAI) 3.901166920 Brown (USA) 0.756854602 Kytola (FIN) 2.118203163 Jeong-Mi (KOR) 3.896847898 or, more conveniently, by extracting the first from all precomputed principal components R> predict(heptathlon_pca)[,1] Joyner-Kersee (USA) -4.757530189 Sablovskaite (URS) -1.288135516 Fleming (AUS) -0.953445060 Bouraga (URS) -0.522322004 Scheider (SWI) 0.003014986 Yuping (CHN) 0.232507119 Mulliner (GB) 1.880932819 Geremias (BRA) 2.770706272 John (GDR) -3.147943402 Choubenkova (URS) -1.503450994 Greiner (USA) -0.633239267 Wijnsma (HOL) -0.217701500 Braun (FRG) 0.109183759 Hagger (GB) 0.659520046 Hautenauve (BEL) 1.828170404 Hui-Ing (TAI) 3.901166920 Behmer (GDR) -2.926184760 Schulz (GDR) -0.958467101 Lajbnerova (CZE) -0.381571974 Dimitrova (BUL) -1.075984276 Ruotsalainen (FIN) 0.208868056 Brown (USA) 0.756854602 Kytola (FIN) 2.118203163 Jeong-Mi (KOR) 3.896847898 The first two components account for 75% of the variance A barplot of each component’s variance (see Figure 16.3) shows how the first two components dominate A plot of the data in the space of the first two principal components, with the points labelled by the name of the corresponding competitor, can be produced as shown with Figure 16.4 In addition, the first two loadings for the events are given in a second coordinate system, also illustrating the special role of the javelin event This graphical representation is known as biplot (Gabriel, 1971) A biplot is a graphical representation of the information in an n × p data matrix The “bi” is a reflection that the technique produces a diagram that gives variance and covariance information about the variables and information about generalised distances between individuals The coordinates used to produce the biplot can all be obtained directly from the principal components analysis of the covariance matrix of the data and so the plots can be viewed as an alternative representation of the results of such an analysis Full details of the technical details of the biplot are given in Gabriel (1981) and in Gower and Hand (1996) Here we simply construct the biplot for the heptathlon data (without PNG); the result is shown in Figure 16.4 The plot clearly shows that the winner of the gold medal, Jackie Joyner-Kersee, accumulates the majority of her points from the three events long jump, hurdles, and 200m © 2010 by Taylor and Francis Group, LLC 294 R> plot(heptathlon_pca) PRINCIPAL COMPONENT ANALYSIS Variances Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 heptathlon_pca Figure 16.3 Barplot of the variances explained by the principal components (with observations for PNG removed) The correlation between the score given to each athlete by the standard scoring system used for the heptathlon and the first principal component score can be found from R> cor(heptathlon$score, heptathlon_pca$x[,1]) [1] -0.9931168 This implies that the first principal component is in good agreement with the score assigned to the athletes by official Olympic rules; a scatterplot of the official score and the first principal component is given in Figure 16.5 © 2010 by Taylor and Francis Group, LLC SUMMARY R> biplot(heptathlon_pca, col = c("gray", "black")) −6 −4 −2 H−In javelin Rtsl Ljbn −2 0.2 0.1 John Chbn Mlln Borg Htnv Bhmr Dmtr Flmn Kytl Jn−M Grnr Schl run200m Sblv hurdles Grms longjump shot Hggr Jy−K Wjns −0.1 0.0 −0.2 PC2 −0.3 highjump −4 Schd Bran −0.4 Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 run800m 295 Ypng Brwn −0.4 −0.2 0.0 0.2 0.4 0.6 PC1 Figure 16.4 Biplot of the (scaled) first two principal components (with observations for PNG removed) 16.4 Summary Principal components look for a few linear combinations of the original variables that can be used to summarise a data set, losing in the process as little information as possible The derived variables might be used in a variety of ways, in particular for simplifying later analyses and providing informative plots of the data The method consists of transforming a set of correlated variables to a new set of variables that are uncorrelated Consequently it should be noted that if the original variables are themselves almost uncorrelated there is little point in carrying out a principal components analysis, since it will merely find components that are close to the original variables but arranged in decreasing order of variance © 2010 by Taylor and Francis Group, LLC −2 −4 heptathlon_pca$x[, 1] Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 296 PRINCIPAL COMPONENT ANALYSIS R> plot(heptathlon$score, heptathlon_pca$x[,1]) 5500 6000 6500 7000 heptathlon$score Figure 16.5 Scatterplot of the score assigned to each athlete in 1988 and the first principal component Exercises Ex 16.1 Apply principal components analysis to the covariance matrix of the heptathlon data (excluding the score variable) and compare your results with those given in the text, derived from the correlation matrix of the data Which results you think are more appropriate for these data? Ex 16.2 The data in Table 16.2 give measurements on five meteorological variables over an 11-year period (taken from Everitt and Dunn, 2001) The variables are year: the corresponding year, rainNovDec: rainfall in November and December (mm), temp: average July temperature, © 2010 by Taylor and Francis Group, LLC SUMMARY 297 Downloaded by [King Mongkut's Institute of Technology, Ladkrabang] at 01:57 11 September 2014 rainJuly: rainfall in July (mm), radiation: radiation in July (curies), and yield: average harvest yield (quintals per hectare) Carry out a principal components analysis of both the covariance matrix and the correlation matrix of the data and compare the results Which set of components leads to the most meaningful interpretation? Table 16.2: meteo data Meteorological measurements in an 11year period year 1920-21 1921-22 1922-23 1923-24 1924-25 1925-26 1926-27 1927-28 1928-29 1929-30 1930-31 rainNovDec 87.9 89.9 153.0 132.1 88.8 220.9 117.7 109.0 156.1 181.5 181.4 temp 19.6 15.2 19.7 17.0 18.3 17.8 17.8 18.3 17.8 16.8 17.0 rainJuly 1.0 90.1 56.6 91.0 93.7 106.9 65.5 41.8 57.4 140.6 74.3 radiation 1661 968 1353 1293 1153 1286 1104 1574 1222 902 1150 yield 28.37 23.77 26.04 25.74 26.68 24.29 28.00 28.37 24.96 21.66 24.37 Source: From Everitt, B S and Dunn, G., Applied Multivariate Data Analysis, 2nd Edition, Arnold, London, 2001 With permission Ex 16.3 The correlations below are for the calculus measurements for the six anterior mandibular teeth Find all six principal components of the data and use a screeplot to suggest how many components are needed to adequately account for the observed correlations Can you interpret the components? Table 16.3: Correlations for calculus measurements for the six anterior mandibular teeth 1.00 0.54 0.34 0.37 0.36 0.62 © 2010 by Taylor and Francis Group, LLC 1.00 0.65 0.65 0.59 0.49 1.00 0.84 0.67 0.43 1.00 0.80 0.42 1.00 0.55 1.00 ... the principal components from the correlation matrix A principal component analysis of the data can be applied using the prcomp function with the scale argument set to TRUE to ensure the analysis. .. September 2014 PRINCIPAL COMPONENT ANALYSIS 287 nient lower-dimensional summary of these variables that might prove useful for a variety of reasons In some applications, the principal components... Scatterplot of the score assigned to each athlete in 1988 and the first principal component Exercises Ex 16.1 Apply principal components analysis to the covariance matrix of the heptathlon data (excluding