Exploratory Data Analysis_8 pot

1. Exploratory Data Analysis 1.3. EDA Techniques 1.3.5. Quantitative Techniques 1.3.5.10.Levene Test for Equality of Variances Purpose: Test for Homogeneity of Variances Levene's test ( Levene 1960) is used to test if k samples have equal variances. Equal variances across samples is called homogeneity of variance. Some statistical tests, for example the analysis of variance, assume that variances are equal across groups or samples. The Levene test can be used to verify that assumption. Levene's test is an alternative to the Bartlett test. The Levene test is less sensitive than the Bartlett test to departures from normality. If you have strong evidence that your data do in fact come from a normal, or nearly normal, distribution, then Bartlett's test has better performance. Definition The Levene test is defined as: H 0 : H a : for at least one pair (i,j). Test Statistic: Given a variable Y with sample of size N divided into k subgroups, where N i is the sample size of the ith subgroup, the Levene test statistic is defined as: where Z ij can have one of the following three definitions: where is the mean of the ith subgroup. 1. where is the median of the ith subgroup. 2. 1.3.5.10. Levene Test for Equality of Variances http://www.itl.nist.gov/div898/handbook/eda/section3/eda35a.htm (1 of 4) [5/1/2006 9:57:20 AM] where is the 10% trimmed mean of the ith subgroup. 3. are the group means of the Z ij and is the overall mean of the Z ij . The three choices for defining Z ij determine the robustness and power of Levene's test. By robustness, we mean the ability of the test to not falsely detect unequal variances when the underlying data are not normally distributed and the variables are in fact equal. By power, we mean the ability of the test to detect unequal variances when the variances are in fact unequal. Levene's original paper only proposed using the mean. Brown and Forsythe (1974)) extended Levene's test to use either the median or the trimmed mean in addition to the mean. They performed Monte Carlo studies that indicated that using the trimmed mean performed best when the underlying data followed a Cauchy distribution (i.e., heavy-tailed) and the median performed best when the underlying data followed a (i.e., skewed) distribution. Using the mean provided the best power for symmetric, moderate-tailed, distributions. Although the optimal choice depends on the underlying distribution, the definition based on the median is recommended as the choice that provides good robustness against many types of non-normal data while retaining good power. If you have knowledge of the underlying distribution of the data, this may indicate using one of the other choices. Significance Level: 1.3.5.10. Levene Test for Equality of Variances http://www.itl.nist.gov/div898/handbook/eda/section3/eda35a.htm (2 of 4) [5/1/2006 9:57:20 AM] Critical Region: The Levene test rejects the hypothesis that the variances are equal if where is the upper critical value of the F distribution with k - 1 and N - k degrees of freedom at a significance level of . In the above formulas for the critical regions, the Handbook follows the convention that is the upper critical value from the F distribution and is the lower critical value. Note that this is the opposite of some texts and software programs. In particular, Dataplot uses the opposite convention. Sample Output Dataplot generated the following output for Levene's test using the GEAR.DAT data set (by default, Dataplot performs the form of the test based on the median): LEVENE F-TEST FOR SHIFT IN VARIATION (CASE: TEST BASED ON MEDIANS) 1. STATISTICS NUMBER OF OBSERVATIONS = 100 NUMBER OF GROUPS = 10 LEVENE F TEST STATISTIC = 1.705910 2. FOR LEVENE TEST STATISTIC 0 % POINT = 0. 50 % POINT = 0.9339308 75 % POINT = 1.296365 90 % POINT = 1.702053 95 % POINT = 1.985595 99 % POINT = 2.610880 99.9 % POINT = 3.478882 90.09152 % Point: 1.705910 3. CONCLUSION (AT THE 5% LEVEL): THERE IS NO SHIFT IN VARIATION. THUS: HOMOGENEOUS WITH RESPECT TO VARIATION. 1.3.5.10. Levene Test for Equality of Variances http://www.itl.nist.gov/div898/handbook/eda/section3/eda35a.htm (3 of 4) [5/1/2006 9:57:20 AM] Interpretation of Sample Output We are testing the hypothesis that the group variances are equal. The output is divided into three sections. The first section prints the number of observations (N), the number of groups (k), and the value of the Levene test statistic. 1. The second section prints the upper critical value of the F distribution corresponding to various significance levels. The value in the first column, the confidence level of the test, is equivalent to 100(1- ). We reject the null hypothesis at that significance level if the value of the Levene F test statistic printed in section one is greater than the critical value printed in the last column. 2. The third section prints the conclusion for a 95% test. For a different significance level, the appropriate conclusion can be drawn from the table printed in section two. For example, for = 0.10, we look at the row for 90% confidence and compare the critical value 1.702 to the Levene test statistic 1.7059. Since the test statistic is greater than the critical value, we reject the null hypothesis at the = 0.10 level. 3. Output from other statistical software may look somewhat different from the above output. Question Levene's test can be used to answer the following question: Is the assumption of equal variances valid? ● Related Techniques Standard Deviation Plot Box Plot Bartlett Test Chi-Square Test Analysis of Variance Software The Levene test is available in some general purpose statistical software programs, including Dataplot. 1.3.5.10. Levene Test for Equality of Variances http://www.itl.nist.gov/div898/handbook/eda/section3/eda35a.htm (4 of 4) [5/1/2006 9:57:20 AM] 1. Exploratory Data Analysis 1.3. EDA Techniques 1.3.5. Quantitative Techniques 1.3.5.11.Measures of Skewness and Kurtosis Skewness and Kurtosis A fundamental task in many statistical analyses is to characterize the location and variability of a data set. A further characterization of the data includes skewness and kurtosis. Skewness is a measure of symmetry, or more precisely, the lack of symmetry. A distribution, or data set, is symmetric if it looks the same to the left and right of the center point. Kurtosis is a measure of whether the data are peaked or flat relative to a normal distribution. That is, data sets with high kurtosis tend to have a distinct peak near the mean, decline rather rapidly, and have heavy tails. Data sets with low kurtosis tend to have a flat top near the mean rather than a sharp peak. A uniform distribution would be the extreme case. The histogram is an effective graphical technique for showing both the skewness and kurtosis of data set. Definition of Skewness For univariate data Y 1 , Y 2 , , Y N , the formula for skewness is: where is the mean, is the standard deviation, and N is the number of data points. The skewness for a normal distribution is zero, and any symmetric data should have a skewness near zero. Negative values for the skewness indicate data that are skewed left and positive values for the skewness indicate data that are skewed right. By skewed left, we mean that the left tail is long relative to the right tail. Similarly, skewed right means that the right tail is long relative to the left tail. Some measurements have a lower bound and are skewed right. For example, in reliability studies, failure times cannot be negative. 1.3.5.11. Measures of Skewness and Kurtosis http://www.itl.nist.gov/div898/handbook/eda/section3/eda35b.htm (1 of 4) [5/1/2006 9:57:21 AM] Definition of Kurtosis For univariate data Y 1 , Y 2 , , Y N , the formula for kurtosis is: where is the mean, is the standard deviation, and N is the number of data points. The kurtosis for a standard normal distribution is three. For this reason, excess kurtosis is defined as so that the standard normal distribution has a kurtosis of zero. Positive kurtosis indicates a "peaked" distribution and negative kurtosis indicates a "flat" distribution. Examples The following example shows histograms for 10,000 random numbers generated from a normal, a double exponential, a Cauchy, and a Weibull distribution. Normal Distribution The first histogram is a sample from a normal distribution. The normal distribution is a symmetric distribution with well-behaved tails. This is indicated by the skewness of 0.03. The kurtosis of 2.96 is near the expected value of 3. The histogram verifies the symmetry. 1.3.5.11. Measures of Skewness and Kurtosis http://www.itl.nist.gov/div898/handbook/eda/section3/eda35b.htm (2 of 4) [5/1/2006 9:57:21 AM] Double Exponential Distribution The second histogram is a sample from a double exponential distribution. The double exponential is a symmetric distribution. Compared to the normal, it has a stronger peak, more rapid decay, and heavier tails. That is, we would expect a skewness near zero and a kurtosis higher than 3. The skewness is 0.06 and the kurtosis is 5.9. Cauchy Distribution The third histogram is a sample from a Cauchy distribution. For better visual comparison with the other data sets, we restricted the histogram of the Cauchy distribution to values between -10 and 10. The full data set for the Cauchy data in fact has a minimum of approximately -29,000 and a maximum of approximately 89,000. The Cauchy distribution is a symmetric distribution with heavy tails and a single peak at the center of the distribution. Since it is symmetric, we would expect a skewness near zero. Due to the heavier tails, we might expect the kurtosis to be larger than for a normal distribution. In fact the skewness is 69.99 and the kurtosis is 6,693. These extremely high values can be explained by the heavy tails. Just as the mean and standard deviation can be distorted by extreme values in the tails, so too can the skewness and kurtosis measures. Weibull Distribution The fourth histogram is a sample from a Weibull distribution with shape parameter 1.5. The Weibull distribution is a skewed distribution with the amount of skewness depending on the value of the shape parameter. The degree of decay as we move away from the center also depends on the value of the shape parameter. For this data set, the skewness is 1.08 and the kurtosis is 4.46, which indicates moderate skewness and kurtosis. Dealing with Skewness and Kurtosis Many classical statistical tests and intervals depend on normality assumptions. Significant skewness and kurtosis clearly indicate that data are not normal. If a data set exhibits significant skewness or kurtosis (as indicated by a histogram or the numerical measures), what can we do about it? One approach is to apply some type of transformation to try to make the data normal, or more nearly normal. The Box-Cox transformation is a useful technique for trying to normalize a data set. In particular, taking the log or square root of a data set is often useful for data that exhibit moderate right skewness. Another approach is to use techniques based on distributions other than the normal. For example, in reliability studies, the exponential, Weibull, and lognormal distributions are typically used as a basis for modeling rather than using the normal distribution. The probability plot 1.3.5.11. Measures of Skewness and Kurtosis http://www.itl.nist.gov/div898/handbook/eda/section3/eda35b.htm (3 of 4) [5/1/2006 9:57:21 AM] correlation coefficient plot and the probability plot are useful tools for determining a good distributional model for the data. Software The skewness and kurtosis coefficients are available in most general purpose statistical software programs, including Dataplot. 1.3.5.11. Measures of Skewness and Kurtosis http://www.itl.nist.gov/div898/handbook/eda/section3/eda35b.htm (4 of 4) [5/1/2006 9:57:21 AM] 1. Exploratory Data Analysis 1.3. EDA Techniques 1.3.5. Quantitative Techniques 1.3.5.12.Autocorrelation Purpose: Detect Non-Randomness, Time Series Modeling The autocorrelation ( Box and Jenkins, 1976) function can be used for the following two purposes: To detect non-randomness in data.1. To identify an appropriate time series model if the data are not random. 2. Definition Given measurements, Y 1 , Y 2 , , Y N at time X 1 , X 2 , , X N , the lag k autocorrelation function is defined as Although the time variable, X, is not used in the formula for autocorrelation, the assumption is that the observations are equi-spaced. Autocorrelation is a correlation coefficient. However, instead of correlation between two different variables, the correlation is between two values of the same variable at times X i and X i+k . When the autocorrelation is used to detect non-randomness, it is usually only the first (lag 1) autocorrelation that is of interest. When the autocorrelation is used to identify an appropriate time series model, the autocorrelations are usually plotted for many lags. 1.3.5.12. Autocorrelation http://www.itl.nist.gov/div898/handbook/eda/section3/eda35c.htm (1 of 4) [5/1/2006 9:57:45 AM] Sample Output Dataplot generated the following autocorrelation output using the LEW.DAT data set: THE LAG-ONE AUTOCORRELATION COEFFICIENT OF THE 200 OBSERVATIONS = -0.3073048E+00 THE COMPUTED VALUE OF THE CONSTANT A = -0.30730480E+00 lag autocorrelation 0. 1.00 1. -0.31 2. -0.74 3. 0.77 4. 0.21 5. -0.90 6. 0.38 7. 0.63 8. -0.77 9. -0.12 10. 0.82 11. -0.40 12. -0.55 13. 0.73 14. 0.07 15. -0.76 16. 0.40 17. 0.48 18. -0.70 19. -0.03 20. 0.70 21. -0.41 22. -0.43 23. 0.67 24. 0.00 25. -0.66 26. 0.42 27. 0.39 28. -0.65 29. 0.03 30. 0.63 31. -0.42 32. -0.36 33. 0.64 34. -0.05 35. -0.60 36. 0.43 37. 0.32 38. -0.64 39. 0.08 40. 0.58 1.3.5.12. Autocorrelation http://www.itl.nist.gov/div898/handbook/eda/section3/eda35c.htm (2 of 4) [5/1/2006 9:57:45 AM] [...]... the normality hypothesis for the normal distribution data set and rejects it for the three non-normal cases Questions The chi-square test can be used to answer the following types of questions: q Are the data from a normal distribution? q Are the data from a log-normal distribution? q Are the data from a Weibull distribution? q Are the data from an exponential distribution? q Are the data from a logistic... the output above Questions The Anderson-Darling test can be used to answer the following questions: q Are the data from a normal distribution? q Are the data from a log-normal distribution? q Are the data from a Weibull distribution? q Are the data from an exponential distribution? q Are the data from a logistic distribution? Importance Many statistical tests and procedures are based on specific distributional... REJECT H0 REJECT H0 The Kolmogorov-Smirnov test can be used to answer the following types of questions: q Are the data from a normal distribution? q Are the data from a log-normal distribution? q Are the data from a Weibull distribution? q Are the data from an exponential distribution? q Are the data from a logistic distribution? http://www.itl.nist.gov/div898/handbook/eda/section3/eda35g.htm (5 of 6) [5/1/2006... Runs Test Case Study The heat flow meter data demonstrate the use of autocorrelation in determining if the data are from a random process The beam deflection data demonstrate the use of autocorrelation in developing a non-linear sinusoidal model Software The autocorrelation capability is available in most general purpose statistical software programs, including Dataplot http://www.itl.nist.gov/div898/handbook/eda/section3/eda35c.htm... Coefficient Plot Case Study Airplane glass failure time data Software The Anderson-Darling goodness-of-fit test is available in some general purpose statistical software programs, including Dataplot http://www.itl.nist.gov/div898/handbook/eda/section3/eda35e.htm (5 of 5) [5/1/2006 9:57:46 AM] 1.3.5.15 Chi-Square Goodness-of-Fit Test 1 Exploratory Data Analysis 1.3 EDA Techniques 1.3.5 Quantitative Techniques... sample of data came from a population with a specific distribution An attractive feature of the chi-square goodness-of-fit test is that it can be applied to any univariate distribution for which you can calculate the cumulative distribution function The chi-square goodness-of-fit test is applied to binned data (i.e., data put into classes) This is actually not a restriction since for non-binned data you... Definition The chi-square test is defined for the hypothesis: H0: Ha: The data follow a specified distribution The data do not follow the specified distribution http://www.itl.nist.gov/div898/handbook/eda/section3/eda35f.htm (1 of 6) [5/1/2006 9:57:46 AM] 1.3.5.15 Chi-Square Goodness-of-Fit Test Test Statistic: For the chi-square goodness-of-fit computation, the data are divided into k bins and the test statistic... for a normal distribution The test statistics show the characteristics of the test; when the data are from a normal distribution, the test statistic is small and the hypothesis is accepted; when the data are from the double exponential, t, and lognormal distributions, the statistics are significant and the hypothesis of an underlying normal distribution is rejected at significance levels of 0.10, 0.05,... ************************************************* ** normal chi-square goodness of fit test y1 ** ************************************************* CHI-SQUARED GOODNESS-OF-FIT TEST NULL HYPOTHESIS H0: DISTRIBUTION FITS THE DATA ALTERNATE HYPOTHESIS HA: DISTRIBUTION DOES NOT FIT THE DATA DISTRIBUTION: NORMAL SAMPLE: NUMBER OF OBSERVATIONS NUMBER OF NON-EMPTY CELLS NUMBER OF PARAMETERS USED = = = TEST: CHI-SQUARED TEST STATISTIC DEGREES... http://www.itl.nist.gov/div898/handbook/eda/section3/eda35f.htm (3 of 6) [5/1/2006 9:57:46 AM] 1.3.5.15 Chi-Square Goodness-of-Fit Test CHI-SQUARED GOODNESS-OF-FIT TEST NULL HYPOTHESIS H0: DISTRIBUTION FITS THE DATA ALTERNATE HYPOTHESIS HA: DISTRIBUTION DOES NOT FIT THE DATA DISTRIBUTION: NORMAL SAMPLE: NUMBER OF OBSERVATIONS NUMBER OF NON-EMPTY CELLS NUMBER OF PARAMETERS USED = = = TEST: CHI-SQUARED TEST STATISTIC DEGREES . 43.0 83 .4167 9.1 783 -4.40 2 75.0 36.4333 4.72 98 8.15 3 2.0 10.4250 2 .87 86 -2.93 4 0.0 2.2603 1.4547 -1.55 5 0.0 0.3973 0.6257 -0.63 6 0.0 0.0 589 0.2424 -0.24 7 0.0 0.0076 0. 086 9 -0.09 8 0.0. 41.7 083 6.4900 -3.65 2 40.0 18. 2167 3.3444 6.51 3 2.0 5.2125 2.0355 -1. 58 4 0.0 1.1302 1.0 286 -1.10 5 0.0 0.1 986 0.4424 -0.45 6 0.0 0.0294 0.1714 -0.17 7 0.0 0.00 38 0.0615 -0.06 8 0.0. 35.0 24.7917 2 .80 83 3.63 3 0.0 6.5750 2.1639 -3.04 4 0.0 1.3625 1.1 186 -1.22 5 0.0 0.2323 0.4777 -0.49 6 0.0 0.0337 0. 183 3 -0. 18 7 0.0 0.0043 0.0652 -0.07 8 0.0 0.0005 0.02 18 -0.02 9 0.0