Ecological validity of neuropsychological tests focus on the california verbal learning test and the wisconsin card sorting test

Archives of Clinical Neuropsychology, Vol 13, No 6, pp 523–534, 1998 Copyright  1998 National Academy of Neuropsychology Printed in the USA All rights reserved 0887-6177/98 $19.00 ϩ 00 PII S0887-6177(97)00038-3 Ecological Validity of Neuropsychological Tests: Focus on the California Verbal Learning Test and the Wisconsin Card Sorting Test Michelle Y Kibby The University of Memphis Maureen Schmitter-Edgecombe Washington State University Charles J Long The University of Memphis To date, few studies have assessed the ecological validity of current measures of memory and executive functioning, particularly the California Verbal Learning Test (CVLT) and the Wisconsin Card Sorting Test (WCST) A battery of tests, including the CVLT and the WCST, was administered to 28 severely head-injured adults who were year or more postinjury Hierarchical linear regression was performed to assess the relationship between these cognitive tests and work functioning while statistically controlling for physical disability, emotional distress, and memory compensation strategies utilized by the participants Results suggest that the CVLT was best at predicting performance on the job; however, the CVLT and the WCST were about equally predictive of the type of position held by a participant Hence, this study demonstrates the ecological validity of measures of memory and executive functioning for predicting work-related skills  1998 National Academy of Neuropsychology Published by Elsevier Science Ltd INTRODUCTION There has been a changing focus in clinical neuropsychology With the availability of increasingly sophisticated neuroradiological procedures, referral questions are now more focused on predicting what a patient can in the everyday world and less on diagnosing and localizing brain impairment Clinical neuropsychologists are being asked to make statements regarding a patient’s functional skills, ability to return to work, and competence in tasks of community living These concerns are what are referred to by the term ecological (Heinrichs, 1990; Sbordone & Long, 1995) The changing referral questions bring up a critical issue: most neuropsychological tests were developed to detect, and attempt to localize, the presence of brain damage rather than to be ecologically sound To date, research establishing the ecologiAddress correspondence to: Michelle Kibby, The University of Memphis, Department of Psychology, Memphis, TN 38152 E-mail: mkibby@msuvx2.memphis.edu 523 524 M Y Kibby et al cal validity of neuropsychological tests has been limited (for a detailed review see Heinrichs, 1990; Long & Kibby, 1995; Puente, 1992; Sbordone & Long, 1995) Neuropsychological tests that were once validated for their ability to detect brain damage now must be evaluated for their utility as ecological instruments The present study investigated the ability of memory and executive functioning measures to predict employment status Memory and executive functioning have been among the most common cognitive constructs studied by researchers investigating the ecological validity of traditional neuropsychological tests This may be because, unlike other cognitive constructs, such as attention, there are widely accepted measures of memory and executive functioning that are normed with demonstrated reliability and validity Researchers have begun to demonstrate the ecological validity of memory tests such as Reitan’s Story Memory Test (Newnan, Heaton, & Lehman, 1978) and the Wechsler Memory Scale (Heaton, Chelune, & Lehman, 1978) For example, tests of short- and long-term memory have been found to be important predictors of return to work (Heaton et al., 1978; Weddell, Oddy, & Jenkins, 1980), of average wage earned on the job (Newnan et al., 1978), and of successful completion of a vocational evaluation (Ryan, Sautter, Capps, Meneese, & Barth, 1992) In addition, memory functioning has also been shown to predict individual’s ability to perform self-care skills or to re-enter the community (Goldstein, McCue, Rogers, & Nussbaum, 1992; Richardson, Nadler, & Malloy, 1995) Traditional measures of executive functioning (e.g., Category Test) have also been shown to be predictive of return to work and of average wage earned The Category Test, in association with other measures, was able to discriminate between employed and chronically unemployed brain-injured persons and to predict average wage earned on the job (Newnan et al., 1978) Nonetheless, in this study, memory functioning better predicted occupational status (type of position held by an individual) than the Category test In a related study, Dikmen and Morgan (1980) found that measures of memory functioning and flexibility in thinking best discriminated between employed and unemployed individuals with epilepsy, although the Category Test gained greater importance with increasing levels of occupational status This is not surprising as all jobs require some level of memory functioning to perform, whereas manual labor positions may require much less executive functioning to perform than higher-level positions While the literature on memory and executive functioning has made strides toward filling the void of research on the ecological validity of neuropsychological tests, there are still two important issues that need to be addressed First, studies have only investigated whether neuropsychological instruments can predict return to work In contrast, little attention has been given to the utility of these measures for predicting how competently an individual can perform on the job Many individuals are able to return to some type of employment after brain injury However, they often return to positions of lower status than they were working at before the injury, or they need additional assistance to continue working at the same type of job Even those that are able to return to the same level position without additional assistance often experience emotional distress related to an inability to perform at the same level as prior to the injury Studies that only investigate whether an individual returns to work are not able to assess how well the person is functioning on the job as compared to before the injury One of the purposes of this study was to investigate how well tests of memory and executive functioning predict competency on the job postinjury (labeled ‘‘job performance’’) A second major issue is that the majority of research conducted in his area has failed to control for relevant factors, such as the presence of physical disability and/or emotional disturbance Both of these factors have been shown to be strong predictors of return to work For example, the presence of motor deficits and other system injuries have been found to be inversely related to return to work (Fraser, Dikmen, McLean, Miller, & Temkin, 1988; Ecological Validity of the CVLT and the WCST 525 Stambrook, Moore, Peter, Deviaene, & Hawryluk, 1990) Physical disability may effect one’s ability to obtain and maintain employment, particularly in the case of laborer positions or positions that entail a great deal of stamina Stambrook et al (1990) also found that patient self-report of depression and psychosocial difficulties were inversely related to postinjury vocational status Failure to take into account such factors may confound the relationship between neuropsychological test data and work functioning In fact, researchers utilizing the clinical scales of the Minnesota Multiphasic Personality Inventory (MMPI) in conjunction with neuropsychological measures have found the combination of neuropsychological and personality measures to be better at predicting return to work than using either measure alone (Heaton et al., 1978; Newnan et al., 1978) The present study investigated the ecological validity of two currently used measures of executive and memory functioning that are appearing more frequently in the literature: the Wisconsin Card Sorting Test (Heaton, 1981) and the California Verbal Learning Test (Delis, Kramer, Kaplan, Ober, & Fridlund, 1987) Both of these instruments combine experimental methodology with a standardized, clinical measure Currently, however, little is still known about the ecological validity of these instruments in terms of predicting occupational status or job performance WISCONSIN CARD SORTING TEST The Wisconsin Card Sorting Test (WCST) is a test of concept formation that evaluates a person’s ability to ‘‘shift set’’ and avoid perseverative responding It is possible for a person to well on the conceptually more difficult Category Test and quite poorly on the WCST because of the latter task’s requirement of shifting response set (Bennett, 1988) Previous research has demonstrated the use of the WCST for diagnosing brain impairment For example, the measure of perseverative responding is especially sensitive to frontal lobe dysfunction (Heaton, 1981) Recent research has also begun to demonstrate its ecological validity The WCST Perserverative Responses significantly correlated with head-injured males’ ability to perform activities of daily living (Little, Templer, Persel, & Ashley, 1996) Lysaker, Bell, and Beam-Goulet (1995) found that the WCST Trials to First Category and Total Number Correct successfully predicted task orientation at a vocational work placement in a sample of persons with schizophrenia Research to date has yet to investigate the utility of the WCST for predicting job performance and occupational status, however CALIFORNIA VERBAL LEARNING TEST The California Verbal Learning Test is a measure of memory for auditorily presented material Research has shown that persons with head injury performed worse on immediate recall, delayed recall, and recognition measures than controls (Crosson, 1988) Persons who had sustained head injuries were also less effective at utilizing internal encoding strategies that may have led to their deficient recall As with most neuropsychological instruments, research has attempted to demonstrate its utility in diagnosing brain impairment, yet much is still unknown as to how this instrument predicts competency in meeting the demands of everyday life This issue is critical, given some of the psychometric limitations that have been raised about the instrument (e.g., inflated norms) (Randolph, Gold, Kozora, Cullum, Hermann, & Wyler, 1994; see Elwood, 1995 for a review) Three factors make this study unique First, the study focuses on a chronic population Neuropsychologists in outpatient settings often see clients with severe head injuries for the first time several months postinjury, and few studies have investigated the ecological validity 526 M Y Kibby et al of neuropsychological tests under such conditions Second, physical disability and emotional distress were assessed so their relationship to return to work could be investigated and statistically controlled The use of memory compensation techniques were also measured, as use of these techniques may act to improve performance on the job despite poor memory functioning and, thereby, influence the relationship between the two Third, this investigation takes evaluating return to work a step further by assessing the qualitative aspects of how an individual is performing on the job (‘‘job performance’’), as well as assessing whether the individual returns to work Based upon the literature review, it is predicted that both the WCST and the CVLT will be predictive of job performance and occupational status when physical disability, emotional distress, and use of memory compensation techniques are taken into account However, it is hypothesized that the CVLT may be a better predictor of job performance than the WCST This hypothesis is based on research by Dikman and Morgan (1980) who found that memory was a strong predictor of employment status in general; executive functioning was predictive only as occupational status increased METHOD Participants This study utilized 28 participants with a diagnosis of closed head injury who were from Memphis, Tennessee and the surrounding area Participants were classified as having a severe head injury if they suffered from posttraumatic amnesia (PTA) of days or more; 89% of the participants reportedly had PTA of over weeks, and 82% had PTA over month PTA was assessed retrospectively by careful clinical questioning of the participant and a significant other Additionally, because we were interested in evaluating the chronic affects of severe closed head injury, only those who were at least year postinjury were studied; 89% of those studied were more than years postinjury, and 57% were more than years postinjury The mean age at testing was 33.20 (SD ϭ 10.00, range ϭ 19.24–54.80 years), and mean education was 13.89 years (SD ϭ 2.061, range ϭ 11–19 years) Fluid intelligence was assessed with the Culture Fair Intelligence Test (M ϭ 88.96, SD ϭ 13.15, range ϭ 50–105) Procedure Participants were tested as part of a larger study (Schmitter-Edgecombe, 1996) The WCST and the CVLT were administered along with other cognitive neuropsychological measures Measures of physical disability, emotional distress, and memory compensation strategies were also obtained Physical disability was measured on a 3-point scale (1 ϭ no disability, ϭ mild to moderate disability, ϭ severe disability) Emotional disturbance was evaluated with the Hopkins Symptom Checklist-58 (Derogatis, Lipman, Rickels, Uhlenhuth, & Covi, 1974) Memory compensation strategies were assessed by use of a checklist where participants rated the frequency with which they used each compensation strategy on a 4-point scale A low score on this measure is indicative of frequent use of memory compensation strategies The possible compensation strategies were based upon those found in Cognitive Rehabilitation Therapy: Late Stages of Recovery (Ylvisaker, Szekeres, Henry, Sullivan, & Wheeler, 1987) WCST and the CVLT From the WCST, the measure of perseverative responses was utilized as it has been found to be the most useful for detecting the presence of brain damage (Heaton, 1981) and to be Ecological Validity of the CVLT and the WCST 527 TABLE Descriptive Data on Independent Variables Analyzed Variable Physical Disability HSCL-58 Memory Strategies WCST PR CVLT Total Recall 1–5 CVLT Short-Delay-Trial CVLT Discriminability M SD Minimum Maximum 1.64 1.46 19.48 23.71 41.96 Ϫ33.11 84.75 83 42 10.08 21.09 13.04 33.67 16.39 1.00 1.00 4.00 4.00 20.00 Ϫ100.00 41.00 3.00 2.48 40.00 75.00 65.00 20.00 100.00 CVLT ϭ California Verbal Learning Test; HSCL ϭ The Hopkins Symptom Check List; M ϭ mean; SD ϭ standard deviation; WCST PR ϭ Wisconsin Card Sorting Test perseverative responses ecologically valid for predicting ADL performance (Little et al., 1996) From the CVLT, immediate, short-delay and long-delay recall measures were selected in order to determine if there are differences in ecological validity between these three types of memory Crosson (1988) found all three to successfully discriminate between individuals with head injury and controls For immediate memory, total number of words recalled on List A Trials 1–5 was used Delayed recall measures are heavily influenced by immediate recall performance In order to assess the decrement in performance from immediate recall to short-delay recall, a formula from the CVLT computerized scoring package was utilized: Short-Delay Free Recall minus Trial Discriminability (the ability to discriminate previously learned words from distracter words after a 20-minute interval) was chosen to represent long-term memory, as it is more independent of immediate recall performance than long-term delayed recall (Crosson, 1988) See Table for descriptive information on the independent variables Job Performance and Occupational Status In terms of job performance, Lam, Priddy, and Johnson (1991) reported a need to expand traditional measures of return to work (i.e., those that state whether or not the individual is employed) to include any difficulties a patient may be experiencing when employed For this purpose, a questionnaire assessing job performance was developed, as such a scale was not found in the literature The 9-point job performance scale is illustrated in the Appendix As this is a newly developed scale, return to work was also ranked on the type of scale used more traditionally by the literature (2 ϭ employed full-time, ϭ employed part-time, and ϭ unemployed) Occupational status was ranked according to the Hollingshead’s (1975) Four Factor Index of Social Status Those who are unemployed were given a score of ‘‘0.’’ Work-related measures were given only to those participants who were working prior to injury (N ϭ 19) in an attempt to reduce the contribution of premorbid factors that may have led to employment problems before the injury Hence, all analyses were conducted on the 19 individuals who were working at the time of the accident See Table for descriptive information on the dependent measures In order to lessen bias introduced by self-report, spouse’s or parent’s ratings of the participant’s job performance were also obtained and compared to those of the participant Research has shown that patients may overstate their functional abilities when compared to objective assessment by nursing personnel (Rubenstein, Schairer, Weiland, & Kane, 1984) Data Analysis Pearson correlations between significant others’ and participants’ ratings on job performance were conducted Hierarchical Linear Regression was performed to investigate the 528 M Y Kibby et al TABLE Descriptive Data on Dependent Measures Variable Job performance a Hours currently working b Hollingshead’s rating c M SD Minimum Maximum 4.16 74 2.00 3.22 81 2.40 1.00 00 00 9.00 2.00 7.00 M ϭ mean; SD ϭ standard deviation a Authors’ 9-point scale developed to assess quality of performance on the job b Traditional scale of return to work: ϭ unemployed, ϭ employed part-time, ϭ employed full-time c Type of position held as assessed by the Hollingshead Occupational Scale relationship between the WCST and CVLT and measures of job performance and occupational status In order to statistically control for physical disability, emotional distress and memory strategies utilized, these variables were entered into the equation in Step (Pedhazur, 1982) The WCST was entered in Step and the CVLT measures in Step The CVLT was entered into the regression equations last, as this position is the most difficult in which to explain additional variance, and, hence, it provides the CVLT with the most conservative test This method was chosen as it was hypothesized that the CVLT would be a better predictor of job performance than the WCST RESULTS Significant Other Ratings Significant others’ ratings correlated significantly with participants’ ratings on job performance (r ϭ 9807, p Ͻ 000) Therefore, analyses were performed only on participants’ ratings Job Performance Versus Return To Work The correlation between the authors’ measure of performance on the job and the traditional measure of return to work was high (r ϭ 8405, p Ͻ 000) However, with the correlation being less than 1.0, there are enough differences between the two measures to justify the need for a measure of quality of performance on the job The scale measuring job performance incorporates the traditional measure of return to work (full- versus part-time versus unemployed) and goes beyond it to measure quality of performance on the job (refer to Appendix) The high correlation between the authors’ scale and the traditional scale demonstrates the new scale’s concurrent validity External validity was demonstrated by utilizing the scale on a sample of normal adults (N ϭ 35) Of the adults, 68.4% achieved a maximum score of Of the 31.6% who did not achieve a 9, some were using additional memory compensation techniques or were having emotional distress on the job Average score for the normal participants on this measure was 7.95 (SD ϭ 1.84), while the mean for the participants with CHI was 4.16 (SD ϭ 3.22) Return to Work When the traditional scale of return to work was used as the dependent measure, none of the analyses using the WCST and the CVLT were significant The only bivariate correlation Ecological Validity of the CVLT and the WCST 529 TABLE Results of Hierarchical Regression Analysis in Predicting Job Performance When CVLT Total Recall Trials 1–5 is Entered Last Into the Equation Variable Step Physical disability Emotional distress Memory strategies Step WCST PR Step CVLT total recall 1–5 Explained Variance (%) B β Ϫ.63 Ϫ2.95 06 Ϫ.16 Ϫ.42 19 1.95 11.69 2.03 04 29 4.24 20 78 34.21* CVLT ϭ California Verbal Learning Test; WCST PR ϭ Wisconsin Card Sorting Test perseverative responses Note Explained variance determined by semipartial correlation F ϭ 4.68 (5, 12); R2 ϭ 66; p ϭ 01 *p Ͻ 01 between the independent variables and the traditional measure to reach significance was the measure of physical disability (r ϭ Ϫ.5686, p ϭ 01) Job Performance When the job performance scale was used as the dependent measure, several important findings emerged Two measures of the CVLT, total number of words recalled on learning trials 1–5 (immediate recall) and Discriminability (long-term recognition memory), were significantly correlated with the authors’ job performance scale The measure of short-delay free-recall failed to reach significance Immediate recall accounted for 34% of the variance in job performance, while recognition memory accounted for 32% of the variance WCST perseverative responses did not significantly correlate with level of job performance in either of the analyses Of the variables entered in Step 1, emotional distress approached significance when in the equation with CVLT immediate recall ( p ϭ 07), and physical disability was significant when in the equation with CVLT Discriminability See Tables and for results of analyses investigating the relationship between the WCST and CVLT and job performance TABLE Results of Hierarchical Regression Analysis in Predicting Job Performance When CVLT Discriminability is Entered Last Into the Equation Variable Step Physical disability Emotional distress Memory strategies Step WCST PR Step CVLT discriminability Explained Variance (%) B β Ϫ1.73 Ϫ2.31 08 Ϫ.45 Ϫ.33 26 06 41 7.11 16 75 32.44** 16.36* 7.25 3.67 CVLT ϭ California Verbal Learning Test; WCST PR ϭ Wisconsin Card Sorting Test perseverative responses Note Explained variance determined by semipartial correlation F ϭ 4.33 (5, 12); R ϭ 64; p ϭ 02 *p Ͻ 05; **p Ͻ 01 530 M Y Kibby et al TABLE Results of Hierarchical Regression Analysis in Predicting Occupational Status When CVLT Total Recall Trials 1–5 is Entered Last Into the Equation Variable Step Physical disability Emotional distress Memory strategies Step WCST PR Step CVLT total recall 1–5 Explained Variance (%) B β Ϫ1.42 Ϫ1.34 03 Ϫ.46 Ϫ.25 13 15.30 4.43 1.09 10 52 16.04 11 54 17.06* CVLT ϭ California Verbal Learning Test; WCST PR ϭ Wisconsin Card Sorting Test perseverative responses Note Explained variance determined by semipartial correlation F ϭ 3.40 (5, 12); R ϭ 59; p ϭ 04 *p Ͻ 05 Occupational Status As forecasted, the WCST perseverative responses predicted occupational status, as measured by the occupational scale from the Hollingshead Four Factor Index of Social Status (Hollingshead, 1975) Correlations between occupational status and WCST perseverative responses approached significance ( p ϭ 052) when in an equation with the CVLT immediate recall measure and reached significance when in an equation with CVLT Discriminability Immediate recall was significant for predicting occupational status, while recognition memory approached significance ( p ϭ 06) Of the variables entered in Step 1, physical disability approached significance when in an equation with CVLT immediate recall ( p ϭ 06) and was significant in an equation with CVLT Discriminabilty See Tables and for results of analyses looking at the relationship between the WCST and CVLT and occupational status DISCUSSION Neuropsychologists often need to make decisions about a patient’s capacities to function in the everyday world They are faced with questions such as, ‘‘is the patient ready to return TABLE Results of Hierarchical Regression Analysis in Predicting Occupational Status When CVLT Discriminability is Entered Last Into the Equation Variable Step Physical disability Emotional distress Memory strategies Step WCST PR Step CVLT Discriminability B β Explained Variance (%) Ϫ1.73 Ϫ1.13 06 Ϫ.56 Ϫ.22 26 25.09* 3.19 3.78 08 67 18.77* 10 57 15.06 CVLT ϭ California Verbal Learning Test; WCST PR ϭ Wisconsin Card Sorting Test perseverative responses Note Explained variance determined by semipartial correlation F ϭ 3.14 (5, 12); R2 ϭ 57; p ϭ 05 *p Ͻ 05 Ecological Validity of the CVLT and the WCST 531 to work or live independently?’’ Many of these neuropsychologists work in outpatient settings where they typically see clients with chronic disorders for the first time months after onset of the disorder Investigating the ecological validity of cognitive neuropsychological tests in a postacute, severely head-injured population has yielded some interesting and surprising results Measures of immediate recall and long-term recognition memory from the CVLT were significant predictors of current level of job performance as measured by the authors’ scale, which combines employment status (full- versus part-time versus unemployed) with quality of performance on the job Immediate recall also correlated significantly with occupational status (i.e., unskilled laborer, professional), and long-term recognition memory approached significance The measure of short-term delayed recall failed to reach significance in any of the equations The ability of immediate recall and recognition memory to predict both job performance and occupational status lends further support to the ecological validity of memory tests in predicting current work functioning In terms of executive functioning, perseverative responses from the WCST were not predictive of job performance In contrast, perseverative thinking was predictive of occupational status This finding concurs with that of Dikmen and Morgan (1980) who found measures of executive functioning to be predictive of employment status only as occupational status increased The hypothesis stating that measures of memory functioning would predict job performance better than the WCST was supported Equations predicting occupational status yielded some interesting findings In these equations, the WCST was not significant when in an equation with immediate recall, but was significant when in an equation with long-term recognition memory (Discriminability) Discriminability was not significant in predicting occupational status but tended toward significance It appears that both memory and perseverative responses are predictive of occupational status and are competing for shared variance This conclusion is supported by the fact that WCST perseverative responses are correlated with CVLT Discriminability (r ϭ Ϫ0.58; p Ͻ 001) and with CVLT immediate recall (r ϭ -0.55; p Ͻ 001) Research has shown that those who not use internal strategies to categorize material at encoding perform worse at retrieval (Crosson, 1988; Kibby, Schmitter-Edgecombe, & Long, 1994) This might reflect the fact that a certain level of executive functioning is necessary in order to most effectively organize and encode new material Indeed, a relationship between measures of retrieval and executive functioning has been demonstrated by Kibby, Schmitter-Edgecombe and Long (1995) Perhaps the overlap between memory and executive functioning helps to explain why the WCST and recognition memory are competing for the same variance when predicting occupational status In terms of variables entered into the first step of the hierarchical linear regression analyses, only physical disability was significant Neither emotional distress nor memory compensation techniques utilized were significant predictors of job performance or of occupational status Emotional distress approached significance for predicting job performance when entered into an equation with immediate recall Stambrook et al (1990) and Newnan et al (1978) found psychosocial variables to be predictive of number of hours working With a larger number of participants, emotional distress may have been significant for predicting performance on the job, which includes the number of hours currently working Physical disability was significant in predicting both job performance and occupational status This is also consistent with findings of Stambrook et al (1990) Having injuries to the body in addition to the brain can make it extremely difficult to return to work, particularly if one holds a position that entails manual labor or high levels of stamina When discussing predicting return to work, Lam et al (1991) stated a need for more comprehensive measures of work functioning Measures of work functioning should go be- 532 M Y Kibby et al yond stating whether a person can return to work to include how the individual is performing on the job The job performance scale included in this study was an attempt at meeting this need The authors’ measure of job performance and the traditional measure of return to work (i.e., full-time employment, part-time employment or unemployed) are highly correlated, yet in this study the cognitive measures were not predictive of return to work alone Perhaps the cognitive measures utilized in this study are better predictors of how well a person can perform on the job than solely of if, and how long, a person can work each day Physical disability, on the other hand, was a significant predictor of return to work This is intuitively logical, as having a physical disability can reduce one’s stamina, making it difficult to work full-time The nonsignificant findings for the WCST and for emotional distress in predicting performance on the job need to be interpreted with caution A small sample size was utilized Occupational status was skewed toward the lower end of the distribution, as there were no individuals from the first category of Hollingshead scale (e.g., physicians, psychologists) The sample studied tended to be of low average IQ; however, this measure probably reflects the affects of the injury and not premorbid status as the average educational level was 13.89 years Finally, the sample was composed of only severely head-injured participants year or more postinjury, which may reduce the findings’ generalizability to acutely injured populations or other disorders Further research is needed on this topic utilizing a larger sample size or different populations of participants In addition, the study should be replicated by testing participants at various times since injury so that the ability of cognitive measures to predict later work performance can be investigated Furthermore, more work needs to be done utilizing scales or tests that measure qualitative aspects of job performance Often persons with head injury are able to return to work part- or full-time, but they have difficulty performing at the same level of competency as they had before the injury Knowledge of how well neuropsychological tests predict a patient’s competency on the job will aid neuropsychologists in determining whether a patient is ready to return to work and if any accommodations are going to be necessary to help the patient succeed This knowledge could be gained through collaborating with vocational rehabilitation Those in vocational rehabilitation have the skills necessary to develop measures that evaluate how well a patient is performing on the job Neuropsychologists could compare how well our test data predict patients’ performance in vocational rehabilitation and patients’ performance on the job once placed By doing this, neuropsychologists will have a heightened awareness of which of our tests are correlated with performance on the job This type of collaboration is essential if ecological research is to expand into predicting job performance Acknowledgments: Portions of this data were presented at the Thirteenth Annual Meeting of the National Academy of Neuropsychology, 1994, Phoenix We gratefully acknowledge the assistance of Amanda Neblett and Betsy Shaver for their help in collecting and scoring the data REFERENCES Bennett, T (1988) Use of Halstead-Reitan Neuropsychological Test Battery in the assessment of head injury Cognitive Rehabilitation, May/June, 18–24 Cattell, R B., & Cattell, A K S (1973) Handbook for the individual or group Culture Fair Intelligence Test: Scale Champaign, IL: IPAT Crosson, B (1988) California Verbal Learning Test (CVLT) performance in severely head-injured and neuropsychologically normal adult males Journal of Clinical and Experimental Neuropsychology, 10, 754–768 Ecological Validity of the CVLT and the WCST 533 Delis, D C., Kramer, J., Kaplan, E., Ober, B A., & Fridlund, A (1987) The California Verbal Learning Test New York: Psychological Corporation Derogatis, L R., Lipman, R S., Rickels, K., Uhlenhuth, E H & Covi, L (1974) The Hopkins Symptoms Checklist (HSCL): A self-report symptom inventory Behavioral Science, 19, 1–15 Dikmen, S., & Morgan, S (1980) Neuropsychological factors related to employability and occupational status in persons with epilepsy Journal of Nervous and Mental Disease, 168, 236–240 Elwood, R W (1995) The California Verbal Learning Test: Psychometric characteristics and clinical application Neuropsychology Review, 5, 173–201 Fraser, R., Dikmen, S., McLean, A., Miller, B., & Temkin, N (1988) Employability of head injury survivors: First year post-injury Rehabilitation Counseling Bulletin, 31, 276–288 Goldstein, G., McCue, M., Rogers, J., & Nussbaum, P D (1992) Diagnostic differences in memory test based predictions of functional capacity in the elderly Neuropsychological Rehabilitation, 2, 307–317 Heaton, R K (1981) A manual for the Wisconsin Card Sorting Test Odessa, FL: Psychological Assessment Resources Heaton, R K., Chelune, G J., & Lehman, R A W (1978) Using neuropsychological tests to assess the likelihood of patient employment Journal of Nervous and Mental Disease, 166, 408–416 Heinrichs, W (1990) Current and emergent applications of neuropsychological assessment: Problems of validity and utility Professional Psychology: Research and Practice, 21, 171–176 Hollingshead, A B (1975) Four Factor Index of Social Status Unpublished manuscript Kibby, M Y., Schmitter-Edgecombe, M E., & Long, C J (1994) The California Verbal Learning Test (CVLT) as a measure of memory functioning in severely head-injured adults and neurologically normal controls Poster session presented at the fourteenth annual meeting of the National Academy of Neuropsychology, Orlando, FL Kibby, M Y., Schmitter-Edgecombe, M E., & Long, C J (1995) Relationships between measures of Auditory Verbal Learning and Executive Functioning in closed head injury Poster session presented at the fifteenth annual meeting of the National Academy of Neuropsychology, San Francisco, CA Lam, C S., Priddy, D A., & Johnson, P (1991) Neuropsychological indicators of employability following traumatic brain injury Rehabilitation Counseling Bulletin, 35, 68–75 Little, A J., Templer, D I., Persel, C S., & Ashley, M J (1996) Feasibility of the neuropsychological spectrum in prediction of outcome following head injury Journal of Clinical Psychology, 52, 455–460 Long, C J., & Kibby, M Y (1995) Ecological validity of neuropsychological tests: A look at neuropsychology’s past and the impact that ecological issues may have on its future Advances in Medical Psychotherapy, 8, 59– 78 Lysaker, P., Bell, M., & Beam-Goulet, J (1995) Wisconsin Card Sorting Test and work performance in schizophrenia Psychiatry Research, 56, 45–51 Newnan, O S., Heaton, R K., & Lehman, R A W (1978) Neuropsychological and MMPI correlates of patients’ future employment characteristics Perceptual and Motor Skills, 46, 625–642 Pedhazur, E J (1982) Multiple regression in behavioral research: Explanation and prediction Fort Worth, TX: Harcourt Brace Jovanovich College Publishers Puente, A E (1992) The status of clinical neuropsychology Archives of Clinical Neuropsychology, 7, 297–312 Randolph, C., Gold, J M., Kozora, E., Cullum, C M., Hermann, B P., & Wyler, A R (1994) Estimating memory function: Disparity of Wechsler Memory Scale-revised and California Verbal Learning Test indices in clinical and normal samples Clinical Neuropsychologist, 8, 99–108 Richardson, E D., Nadler, J D., & Malloy, P F (1995) Neuropsychologic prediction of performance measures of daily living skills in geriatric patients Neuropsychology, 9, 565–572 Rubenstein, L Z., Schairer, C., Weiland, G D., & Kane, R (1984) Systematic biases in functional status assessment of elderly adults: Effects of different data sources Journal of Gerontology, 39, 686–691 Ryan, T V., Sautter, S W., Capps, C F., Meneese, & Barth, J T (1992) Utilizing neuropsychological measures to predict vocational outcome in a head trauma population Brain Injury, 6, 175–182 Sbordone, R J., & Long, C J (Eds.) (1995) The ecological validity of neuropsychological testing Delray Beach, FL: St Lucie Press Schmitter-Edgecombe, M (1996) Effects of divided attention on implicit and explicit memory performance following severe closed head injury Neuropsychology, 10, 155–167 Stambrook, M Moore, A D., Peter, L C., Deviaene, C., & Hawryluk, G A (1990) Effects of mild, moderate and severe closed head injury on long-term vocational status Brain Injury, 4, 183–190 Weddell, R., Oddy, M., & Jenkins, D (1980) Social adjustment after rehabilitation: A two year follow-up of patients with severe head injury Psychological Medicine, 10, 257–263 Ylvisaker, M., Szekeres, S F., Henry, K., Sullivan, D M., & Wheeler, P (1987) Topics in cognitive rehabilitation therapy In M Ylvisaker & E M R Gobble (eds.), Community re-entry for head injured adults (pp 137–220) Boston: College-Hill Press 534 M Y Kibby et al APPENDIX Job Performance Scale Say to the participant ‘‘tell me the one number that best describes what you are doing right now.’’ If the individual is also going to school so he or she is not working full time, please note this on the bottom of the page 1) not working because 2) working in a highly structured environment (sheltered workshop) 3) working under close supervision or in a supported work environment Tasks are given with detailed instructions, and the participant is checked regularly to see how he/she is performing on the task 4) working without much supervision beyond what non-head injury survivors need, but with decreased quality of work The participant is having difficulty initiating activities, completing tasks or relating to others 5) working without much supervision beyond what non-head injury survivors need with fairly good quality with the aid of compensation strategies (detailed lists of things to or how to them, memory aids) OR working independently with a reduced load (some work is given to other people to do) or with modified job tasks (tasks are made simpler for the individual) 6) working at a comparable level to non-head injury survivors, but at a lower level job than before the injury 7) working at a comparable level to non-head injury survivors at the same job level as before the injury, but with reduced hours compared to before the injury (not due to school) 8) working at the same type of job as before the injury at the same number of hours and at a comparable level to non-head injury survivors; however, the participant has fatigue, headaches, depression, memory or concentration problems He/She is not using extra compensation strategies or in need of special assistance, but is having difficulty with the job 9) working at the same type of job as before at the same number of hours and with the same level of ability as before the injury, but without the difficulty and the complaints mentioned above in #8  ... more frequently in the literature: the Wisconsin Card Sorting Test (Heaton, 1981) and the California Verbal Learning Test (Delis, Kramer, Kaplan, Ober, & Fridlund, 1987) Both of these instruments... performance WISCONSIN CARD SORTING TEST The Wisconsin Card Sorting Test (WCST) is a test of concept formation that evaluates a person’s ability to ‘‘shift set’’ and avoid perseverative responding It is... investigate the utility of the WCST for predicting job performance and occupational status, however CALIFORNIA VERBAL LEARNING TEST The California Verbal Learning Test is a measure of memory for