Exploratory Data Analysis_16 doc

I STAT EXP(STAT) SD(STAT) Z 1 132.0 208.4167 14.5370 -5.26 2 66.0 91.4333 7.4947 -3.39 3 28.0 26.2583 4.5674 0.38 4 10.0 5.7127 2.3123 1.85 5 6.0 1.0074 0.9963 5.01 6 7.0 0.1498 0.3866 17.72 7 3.0 0.0193 0.1389 21.46 8 1.0 0.0022 0.0468 21.30 9 0.0 0.0002 0.0150 -0.01 10 1.0 0.0000 0.0045 220.19 STATISTIC = NUMBER OF RUNS TOTAL OF LENGTH I OR MORE I STAT EXP(STAT) SD(STAT) Z 1 254.0 333.0000 9.4110 -8.39 2 122.0 124.5833 6.2868 -0.41 3 56.0 33.1500 4.8561 4.71 4 28.0 6.8917 2.5154 8.39 5 18.0 1.1790 1.0761 15.63 6 12.0 0.1716 0.4136 28.60 7 5.0 0.0217 0.1474 33.77 8 2.0 0.0024 0.0494 40.43 9 1.0 0.0002 0.0157 63.73 10 1.0 0.0000 0.0047 210.77 LENGTH OF THE LONGEST RUN UP = 10 LENGTH OF THE LONGEST RUN DOWN = 7 LENGTH OF THE LONGEST RUN UP OR DOWN = 10 NUMBER OF POSITIVE DIFFERENCES = 258 NUMBER OF NEGATIVE DIFFERENCES = 241 NUMBER OF ZERO DIFFERENCES = 0 Values in the column labeled "Z" greater than 1.96 or less than -1.96 are statistically significant at the 5% level. Numerous values in this column are much larger than +/-1.96, so we conclude that the data are not random. Distributional Assumptions Since the quantitative tests show that the assumptions of randomness and constant location and scale are not met, the distributional measures will not be meaningful. Therefore these quantitative tests are omitted. 1.4.2.3.2. Test Underlying Assumptions http://www.itl.nist.gov/div898/handbook/eda/section4/eda4232.htm (7 of 7) [5/1/2006 9:58:36 AM] 1. Exploratory Data Analysis 1.4. EDA Case Studies 1.4.2. Case Studies 1.4.2.3. Random Walk 1.4.2.3.3.Develop A Better Model Lag Plot Suggests Better Model Since the underlying assumptions did not hold, we need to develop a better model. The lag plot showed a distinct linear pattern. Given the definition of the lag plot, Y i versus Y i-1 , a good candidate model is a model of the form Fit Output A linear fit of this model generated the following output. LEAST SQUARES MULTILINEAR FIT SAMPLE SIZE N = 499 NUMBER OF VARIABLES = 1 NO REPLICATION CASE PARAMETER ESTIMATES (APPROX. ST. DEV.) T VALUE 1 A0 0.501650E-01 (0.2417E-01) 2.075 2 A1 YIM1 0.987087 (0.6313E-02) 156.4 RESIDUAL STANDARD DEVIATION = 0.2931194 RESIDUAL DEGREES OF FREEDOM = 497 The slope parameter, A1, has a t value of 156.4 which is statistically significant. Also, the residual standard deviation is 0.29. This can be compared to the standard deviation shown in the summary table, which is 2.08. That is, the fit to the autoregressive model has reduced the variability by a factor of 7. Time Series Model This model is an example of a time series model. More extensive discussion of time series is given in the Process Monitoring chapter. 1.4.2.3.3. Develop A Better Model http://www.itl.nist.gov/div898/handbook/eda/section4/eda4233.htm (1 of 2) [5/1/2006 9:58:36 AM] 1.4.2.3.3. Develop A Better Model http://www.itl.nist.gov/div898/handbook/eda/section4/eda4233.htm (2 of 2) [5/1/2006 9:58:36 AM] 1. Exploratory Data Analysis 1.4. EDA Case Studies 1.4.2. Case Studies 1.4.2.3. Random Walk 1.4.2.3.4.Validate New Model Plot Predicted with Original Data The first step in verifying the model is to plot the predicted values from the fit with the original data. This plot indicates a reasonably good fit. Test Underlying Assumptions on the Residuals In addition to the plot of the predicted values, the residual standard deviation from the fit also indicates a significant improvement for the model. The next step is to validate the underlying assumptions for the error component, or residuals, from this model. 1.4.2.3.4. Validate New Model http://www.itl.nist.gov/div898/handbook/eda/section4/eda4234.htm (1 of 4) [5/1/2006 9:58:40 AM] 4-Plot of Residuals Interpretation The assumptions are addressed by the graphics shown above: The run sequence plot (upper left) indicates no significant shifts in location or scale over time. 1. The lag plot (upper right) exhibits a random appearance.2. The histogram shows a relatively flat appearance. This indicates that a uniform probability distribution may be an appropriate model for the error component (or residuals). 3. The normal probability plot clearly shows that the normal distribution is not an appropriate model for the error component. 4. A uniform probability plot can be used to further test the suggestion that a uniform distribution might be a good model for the error component. 1.4.2.3.4. Validate New Model http://www.itl.nist.gov/div898/handbook/eda/section4/eda4234.htm (2 of 4) [5/1/2006 9:58:40 AM] Uniform Probability Plot of Residuals Since the uniform probability plot is nearly linear, this verifies that a uniform distribution is a good model for the error component. Conclusions Since the residuals from our model satisfy the underlying assumptions, we conlude that where the E i follow a uniform distribution is a good model for this data set. We could simplify this model to This has the advantage of simplicity (the current point is simply the previous point plus a uniformly distributed error term). Using Scientific and Engineering Knowledge In this case, the above model makes sense based on our definition of the random walk. That is, a random walk is the cumulative sum of uniformly distributed data points. It makes sense that modeling the current point as the previous point plus a uniformly distributed error term is about as good as we can do. Although this case is a bit artificial in that we knew how the data were constructed, it is common and desirable to use scientific and engineering knowledge of the process that generated the data in formulating and testing models for the data. Quite often, several competing models will produce nearly equivalent mathematical results. In this case, selecting the model that best approximates the scientific understanding of the process is a reasonable choice. 1.4.2.3.4. Validate New Model http://www.itl.nist.gov/div898/handbook/eda/section4/eda4234.htm (3 of 4) [5/1/2006 9:58:40 AM] Time Series Model This model is an example of a time series model. More extensive discussion of time series is given in the Process Monitoring chapter. 1.4.2.3.4. Validate New Model http://www.itl.nist.gov/div898/handbook/eda/section4/eda4234.htm (4 of 4) [5/1/2006 9:58:40 AM] 1. Exploratory Data Analysis 1.4. EDA Case Studies 1.4.2. Case Studies 1.4.2.3. Random Walk 1.4.2.3.5.Work This Example Yourself View Dataplot Macro for this Case Study This page allows you to repeat the analysis outlined in the case study description on the previous page using Dataplot . It is required that you have already downloaded and installed Dataplot and configured your browser. to run Dataplot. Output from each analysis step below will be displayed in one or more of the Dataplot windows. The four main windows are the Output window, the Graphics window, the Command History window, and the data sheet window. Across the top of the main windows there are menus for executing Dataplot commands. Across the bottom is a command entry window where commands can be typed in. Data Analysis Steps Results and Conclusions Click on the links below to start Dataplot and run this case study yourself. Each step may use results from previous steps, so please be patient. Wait until the software verifies that the current step is complete before clicking on the next step. The links in this column will connect you with more detailed information about each analysis step from the case study description. 1. Invoke Dataplot and read data. 1. Read in the data. 1. You have read 1 column of numbers into Dataplot, variable Y. 2. Validate assumptions. 1. 4-plot of Y. 2. Generate a table of summary statistics. 3. Generate a linear fit to detect drift in location. 4. Detect drift in variation by dividing the data into quarters and computing Levene's test for equal 1. Based on the 4-plot, there are shifts in location and scale and the data are not random. 2. The summary statistics table displays 25+ statistics. 3. The linear fit indicates drift in location since the slope parameter is statistically significant. 4. Levene's test indicates significant drift in variation. 1.4.2.3.5. Work This Example Yourself http://www.itl.nist.gov/div898/handbook/eda/section4/eda4235.htm (1 of 2) [5/1/2006 9:58:40 AM] standard deviations. 5. Check for randomness by generating a runs test. 5. The runs test indicates significant non-randomness. 3. Generate the randomness plots. 1. Generate an autocorrelation plot. 2. Generate a spectral plot. 1. The autocorrelation plot shows significant autocorrelation at lag 1. 2. The spectral plot shows a single dominant low frequency peak. 4. Fit Y i = A0 + A1*Y i-1 + E i and validate. 1. Generate the fit. 2. Plot fitted line with original data. 3. Generate a 4-plot of the residuals from the fit. 4. Generate a uniform probability plot of the residuals. 1. The residual standard deviation from the fit is 0.29 (compared to the standard deviation of 2.08 from the original data). 2. The plot of the predicted values with the original data indicates a good fit. 3. The 4-plot indicates that the assumptions of constant location and scale are valid. The lag plot indicates that the data are random. However, the histogram and normal probability plot indicate that the uniform disribution might be a better model for the residuals than the normal distribution. 4. The uniform probability plot verifies that the residuals can be fit by a uniform distribution. 1.4.2.3.5. Work This Example Yourself http://www.itl.nist.gov/div898/handbook/eda/section4/eda4235.htm (2 of 2) [5/1/2006 9:58:40 AM] 1. Exploratory Data Analysis 1.4. EDA Case Studies 1.4.2. Case Studies 1.4.2.4.Josephson Junction Cryothermometry Josephson Junction Cryothermometry This example illustrates the univariate analysis of Josephson junction cyrothermometry. Background and Data1. Graphical Output and Interpretation2. Quantitative Output and Interpretation3. Work This Example Yourself4. 1.4.2.4. Josephson Junction Cryothermometry http://www.itl.nist.gov/div898/handbook/eda/section4/eda424.htm [5/1/2006 9:58:48 AM] [...]... AM] 1.4.2.5.1 Background and Data 1 Exploratory Data Analysis 1.4 EDA Case Studies 1.4.2 Case Studies 1.4.2.5 Beam Deflections 1.4.2.5.1 Background and Data Generation This data set was collected by H S Lew of NIST in 1969 to measure steel-concrete beam deflections The response variable is the deflection of a beam from the center point The motivation for studying this data set is to show how the underlying... motivation for studying this data set is to illustrate the case where there is discreteness in the measurements, but the underlying assumptions hold In this case, the discreteness is due to the data being integers This file can be read by Dataplot with the following commands: SKIP 25 SET READ FORMAT 5F5.0 SERIAL READ SOULEN.DAT Y SET READ FORMAT Resulting Data The following are the data used for this case... Example Yourself 1 Exploratory Data Analysis 1.4 EDA Case Studies 1.4.2 Case Studies 1.4.2.4 Josephson Junction Cryothermometry 1.4.2.4.4 Work This Example Yourself View Dataplot Macro for this Case Study This page allows you to repeat the analysis outlined in the case study description on the previous page using Dataplot It is required that you have already downloaded and installed Dataplot and configured... browser to run Dataplot Output from each analysis step below will be displayed in one or more of the Dataplot windows The four main windows are the Output window, the Graphics window, the Command History window, and the data sheet window Across the top of the main windows there are menus for executing Dataplot commands Across the bottom is a command entry window where commands can be typed in Data Analysis...1.4.2.4.1 Background and Data 1 Exploratory Data Analysis 1.4 EDA Case Studies 1.4.2 Case Studies 1.4.2.4 Josephson Junction Cryothermometry 1.4.2.4.1 Background and Data Generation This data set was collected by Bob Soulen of NIST in October, 1971 as a sequence of observations collected equi-spaced in time... affected by periodic data This file can be read by Dataplot with the following commands: SKIP 25 READ LEW.DAT Y Resulting Data The following are the data used for this case study -213 -564 -35 -15 141 115 -420 -360 203 -338 -431 194 -220 -513 154 -125 -559 92 -21 -579 http://www.itl.nist.gov/div898/handbook/eda/section4/eda4251.htm (1 of 6) [5/1/2006 9:58:50 AM] 1.4.2.5.1 Background and Data -52 99 -543... information about each analysis step from the case study description 1 Invoke Dataplot and read data 1 Read in the data 1 You have read 1 column of numbers into Dataplot, variable Y 2 4-plot of the data 1 4-plot of Y 1 Based on the 4-plot, there are no shifts in location or scale Due to the nature of the data (a few distinct points with many repeats), the normality assumption is questionable 3 Generate the individual... http://www.itl.nist.gov/div898/handbook/eda/section4/eda4244.htm (2 of 2) [5/1/2006 9:58:50 AM] 1.4.2.5 Beam Deflections 1 Exploratory Data Analysis 1.4 EDA Case Studies 1.4.2 Case Studies 1.4.2.5 Beam Deflections Beam Deflection This example illustrates the univariate analysis of beam deflection data 1 Background and Data 2 Test Underlying Assumptions 3 Develop a Better Model 4 Validate New Model 5 Work This Example Yourself... in the data 3 The histogram (lower left) shows that the data are reasonably symmetric, there does not appear to be significant outliers in the tails, and that it is reasonable to assume that the data can be fit with a normal distribution 4 The normal probability plot (lower right) is difficult to interpret due to the fact that there are only a few distinct values with many repeats The integer data with... 1.4.2.4.3 Quantitative Output and Interpretation 1 Exploratory Data Analysis 1.4 EDA Case Studies 1.4.2 Case Studies 1.4.2.4 Josephson Junction Cryothermometry 1.4.2.4.3 Quantitative Output and Interpretation Summary Statistics As a first step in the analysis, a table of summary statistics is computed from the data The following table, generated by Dataplot, shows a typical set of statistics SUMMARY . detailed information about each analysis step from the case study description. 1. Invoke Dataplot and read data. 1. Read in the data. 1. You have read 1 column of numbers into Dataplot, variable Y. 2 [5/1/2006 9:58:48 AM] 1. Exploratory Data Analysis 1.4. EDA Case Studies 1.4.2. Case Studies 1.4.2.4. Josephson Junction Cryothermometry 1.4.2.4.1.Background and Data Generation This data set was collected. the data sheet window. Across the top of the main windows there are menus for executing Dataplot commands. Across the bottom is a command entry window where commands can be typed in. Data Analysis