EXPERTISE AND AGING IN A PILOT DECISION-MAKING TASK

Daniel Morrow¹, Lisa Soederberg Miller², Heather Ridolfo², Nina Kokayeff¹, Dervon Chang¹,

Ute Fischer³, and Elizabeth Stine-Morrow¹

1. University of Illinois at Urbana-Champaign; 2. Brandeis University;

3. Georgia Institute of Technology

We examined age/expertise trade-offs in a laboratory pilot decision-making task. Expert and novice pilots read at their own pace brief scenarios that described simpler or more complex flight situations, then in a standard interview discussed the problem in the scenario and how they would respond if they were pilot-in-command. Decision making was measured by coding the protocols for correctly identifying the problems and solutions to problems. Scenario comprehension was measured by reading time and the accuracy of answering questions about the scenarios. All groups accurately identified the problems, but experts elaborated problem descriptions more than novices did. Older experts elaborated more, and older novices elaborated less, than their younger counterparts. Older experts also identified more appropriate solutions to problems while older novices identified less appropriate solutions compared to their younger counterparts. Reading time findings suggested that experts maintained decision-making accuracy by spending more time on critical information when reading more complex scenarios.

INTRODUCTION

Expert Decision Making

Decision making in many domains involves identifying problems and determining appropriate solutions to these problems in the face of complex, ill-defined and dynamic conditions (Klein, 1993; Orasanu & Fischer, 1997). Processes such as cue integration and evaluating alternative solutions can place heavy demands on working memory (Wickens & Hollands, 2000). Decision making in real-world domains is also supported by expertise. Experts tend to more accurately assess the problem situation and retrieve appropriate solutions based on this assessment. These benefits reflect organized domain knowledge that supports comprehension and decision-making (Ericsson & Kintsch, 1995; Klein, 1993). Aviation expertise benefits have been found for several decision-making tasks (Wickens, Stokes, Barnett, & Hyman, 1993; O’Hare, 2003), in part reflecting attention allocation to relevant cues guided by an accurate situation model of the flight situation (Bellenkes, Wickens, & Kramer, 1997; O'Hare, 2003). Thus, expert decision making rests in part on the ability to create a mental model representing critical aspects of current and possible future states of the situation (Adams, Tenney, & Pew, 1995). Although comprehension and decision-making processes are linked in expertise theories, few studies have examined expertise effects on both comprehension and decision making. We investigated the impact of pilot expertise and aging on understanding and making decisions about flight-related situations.

Aging, Comprehension, and Decision Making

The impact of aging on decision making may depend on trade-offs between cognitive demands and knowledge relevant to the decision-making task. On the one hand, normal aging is associated with gradual declines in working memory capacity, processing speed, and other basic cognitive abilities, both in the general population (Salthouse, 1991) and among expert pilots (e.g., Morrow, Ridolfo et al., 2003). If cue integration and other decision-making processes tax working memory, older pilots may be disadvantaged compared to younger pilots. On the other hand, age-related cognitive changes may have little impact on decision making if expertise facilitates these processes and reduces demands on working memory. Older pilots tend to have more flying experience and only modest declines on declarative knowledge measures (Morrow, Ridolfo et al., 2003). While aging effects on pilot decision making has received little attention, Stokes et al. (1990) found that experts’ decisions during simulated flight were less influenced than novices by time-related stress, suggesting expertise can compensate for cognitive demands imposed by stress. Moreover, age-related differences on pilot communication tasks are mitigated when the task reduces demands on cognitive resources and supports the use of domain knowledge (Morrow, Ridolfo et al., 2003). Expertise may especially benefit older pilot judgment and decision-making if older pilots can rely on experience relevant to the problem (Klein, 1993; Mohler, 1981).

Additional insight into trade-offs between knowledge and cognitive resource demands on decision making can be gleaned from the literature on aging and text comprehension. Older adults are at least as likely as younger adults to use knowledge to create situation models when reading (Stine-Morrow, Morrow, & Leno, 2002). Moreover, younger adults tend to focus more on the text-base level of processing when first reading a text and shift to situation model processing when re-reading the same text, but older adults focus on the situation model when first reading the text (Stine-Morrow, et al., 2004). While knowledge-based processes such as inferencing require investment of cognitive resources, older adults sometimes allocate resources to knowledge use during comprehension (Miller, 2003). In other words, age-related differences in resource allocation to knowledge use during comprehension and decision making may offset the impact of cognitive declines. However, age-related knowledge benefits may depend on the cognitive demands associated with creating a situation model. More cognitive effort may be required to use knowledge to interpret situations with complex problems and to evaluate solutions to these problems (Klein, 1993). Age-related cognitive declines may limit older experts’ ability to draw upon domain knowledge to understand and respond to more complex situations.

Predictions

We examined age/expertise trade-offs in a laboratory pilot decision-making task. Expert and novice pilots read brief scenarios that described simpler or more complex flight situations and then discussed the problem in the scenario and how they would respond if they were pilot-in-command. Decision making was measured by coding the protocols for identifying the problems and solutions to problems. Comprehension was measured by reading time and accuracy of answering questions about the scenarios. We expected older expert pilots to spend as much or more time as younger experts reading the scenarios in order to develop a situation model that supports similar levels of decision-making accuracy. Novice pilots should be less accurate in scenario comprehension and decision making, with greater age-related declines for this group.

METHODS

Participants

Expert pilots (airline and corporate) and novice pilots (General Aviation without commercial experience) participated. Nineteen older (46-60 years) and 20 younger (23-42 years) experts and 13 older and 25 younger novices participated. All older experts and seven younger experts served as captains at the time of the study. The other 13 younger experts were first officers. Table 1 shows that experts and novices did not differ on measures of education, working memory, processing speed, or vocabulary. Older experts as well as novices showed typical age-related declines for the working memory and speed measures and an age-related increase in vocabulary. Age did not interact with expertise on these measures, showing that both groups had similar age-related cognitive profiles. Expertise was measured by:

a) flight hours, b) a general test of aviation knowledge adapted from an FAA exam, and c) scenario-specific knowledge questionnaire that assessed concepts relevant to complex commercial operations. The latter two declarative knowledge measures were included because expertise is only loosely related to amount of experience (Ericsson & Charness, 1994). Table 1 shows that, not surprisingly, experts had flown more hours. Both groups showed an age-related increase in total flight experience, although this increase was smaller for novices. Experts outperformed novices on the declarative knowledge measures (marginally significant difference for the general expertise measure), which helps validate the qualitative expert/novice distinction based on type of flying experience (commercial versus GA).

Table 1

Mean Demographic and Cognitive Ability Scores

	Yng Exprt N=20	Old Exprt N=19	M	Yng Nvice N=25		Old Nvice N=13	M		Age F(1,73)		Exprt F(1,73)		Age x Exprt
Age	32.6	54.9	43.7		26.0	54.2		40.1		377.6*		7.8*		5.2*
Educ	15.9	16.6	16.2		15.6	16.3		15.9		2.7		<1.0		<1.0
WM^1	4.6	4.1	4.4		4.3	3.9		4.1		4.7*		2.0		<1.0
Speed^2	30.6	28.2	29.4		31.0	26.7		28.9		14.8*		<1.0		1.2
Voc^3	16.9	22.2	19.5		14.6	22.4		18.5		40.4*		1.0		1.4
Total Flight Hrs	6247	14399	10323		457	647		552		33.4*		183.2*		30.4*
Hrs last 12 months	620	606	613		172	54		149		2.3		131.6*		1.4
Total IFR Hrs	1667	4232	2949		81	104		86		3.1		14.9*		3.0
AK: Genral^4	15.5	14.3	14.9		14.5	13.3		13.9		4.8*		3.3		<1.0
AK: Specf^5	11.0	11.2	11.1		10.0	10.2		10.1		<1.0		4.9*		<1.0

* p <.05

1. Mean of listening and reading versions of the sentence span task (correlation of the two tests=.36**), a measure of verbal working memory capacity (Stine & Hindman, 1994).

2. Mean of Letter and Pattern Comparison tasks (correlation of the two tests=.38**), a measure of processing speed (Salthouse & Babcock, 1991).

3. Advanced Vocabulary Test from the Kit of Factor-Referenced Cognitive tests (Ekstrom, French, & Harmon, 1976)

4. General aviation knowledge measure: Twenty-item test adapted from FAA commercial pilot’s license examination.

5. Specific aviation knowledge measure: Twelve-item questionnaire developed by the pilots who developed the scenarios. Administered to 49 participants (15 experts and 34 novices).

Procedure and Scenarios

Participants read six brief scenarios that described simple or more complex flight situations involving commercial aircraft, with which the experts should be more familiar. Each scenario consisted of a “set-up” and a narrative section. The set-up was a list of categories representing key dimensions of the scenario’s setting, such as departure and destination airport, position of the aircraft, wind, temperature, and weather conditions at the time of the scenario, and type of aircraft. Thus, this section provided a framework for understanding the subsequent narrative. The latter was presented in paragraph form and described a specific situation that occurred during the take-off, enroute, or approach phase of a flight by a complex commercial aircraft. Each scenario had simple and complex versions roughly equated for text-base characteristics (average length=152 words). A counterbalance scheme ensured that each participant read three simple and three complex scenarios and that each version of every scenario was equally likely to occur across participants. The set-up sections of the simple and complex scenarios were identical, but the narratives of the complex scenarios described more multi-faceted problems with less clear-cut solutions, and were thought to require more knowledge about aircraft systems and operations. For example, the simpler version of one scenario described an aircraft’s wing striking a crane on take-off, with no apparent problems; in the more complex version the strike resulted in loss of hydraulic pressure and leading edge device asymmetry (ratings by airline pilots in a preliminary study helped validate the simple/complex distinction).

Participants were instructed to read each scenario two times at their own pace. Both times, scenarios were presented on a computer screen, one clause at a time. Then in a standardized interview, they discussed the problem and how they would respond if they were pilot-in-command. They then answered comprehension questions about the scenario.

Two older airline pilots (who also developed the scenarios) and one younger airline pilot agreed on the problem and most appropriate solution to the problem for each scenario. They also identified a set of factors relevant to each scenario’s problem and solution. These were included to measure the extent to which participants elaborated their description of the problem and solution, which might reflect more differentiated situation models. Two pilots used this scheme to independently code a sample of 24 protocols, with 97% inter-rater reliability.

RESULTS

Decision Making

Percent correctly identified problems, extent of problem elaboration (percent of factors mentioned in protocol), percent correctly identified responses, and extent of solution elaboration were analyzed by an Expertise x Age x Scenario Complexity ANOVA, with the latter variable a repeated measure. All groups accurately identified the problems (Expert=96%, Novice=93% correct, F(1,73)=1.8, p > .10; Younger = 96%, Older=93%, F(1,73)=1.6, p > .10). Simple scenario problems were identified more accurately than complex scenario problems (97% versus 93%), F(1,73)=4.8, p <.05, but complexity did not interact with age or expertise. Experts had more elaborate problem descriptions than novices did, F(1,73)=7.8, p < .01 (Figure 1). However, older experts elaborated more, and older novices elaborated less than their younger counterparts, Age X Expertise F(1,73)=8.4, p < .01. Complex problems were elaborated more than simple problems, F(1,73)=10.9, p < .01, suggesting complex scenarios required more domain knowledge. Complexity did not interact with age or expertise, all F’s < 1.0.

Experts also identified more appropriate solutions to problems, F(1,73)=4.5, p < .05 (Figure 2). A marginally significant Age X Expertise interaction, F(1,73)=3.7, p < .06, suggested that older experts were more accurate while older novices were less accurate than their younger counterparts. Finally, experts had more elaborate solution descriptions (30% versus 21%), F(1,73)=9.9, p < .01, but elaboration did not differ by age group (Y=23%, O=24%) or scenario complexity (simple=26%, complex= 27%), F(1,73) < 1.0 for both.

It is possible that some pilots would have an advantage on these decision making measures (especially on elaboration variables) simply by talking more during the protocol, so that group differences would reflect differences in communication style rather than decision accuracy. However, the pattern of findings for the decision-making analyses did not substantially change when mean protocol length was entered into the analyses as a covariate (e.g., problem elaboration Age x Expertise interaction was still significant, F(1,73)=8.5, p < .01).

Figure 1.

Figure 2.

Scenario Comprehension

Reading time was analyzed to examine expertise differences in comprehension processes and whether they related to comprehension and decision making accuracy. As in earlier studies (Stine-Morrow et al., 2004), regression analyses were used to investigate allocation of reading time to different features of the text, which would suggest attention to different levels of representation. Situation model processing was indexed by expert pilot ratings of how critical each clause was to identifying the problem and/or solution to the problem in the scenario (Critical Region variable). On average, there were 7.2 critical regions across the simple versions of the scenarios, and 7.7 critical regions across the complex versions. More superficial, or text-level, processing was indexed by clause length in syllables (other text variables were measured but dropped from the analysis because of co-linearity effects). Individual subjects’ standardized regression coefficients were analyzed by an Age x Expertise x Scenario Complexity x Trial (first or second reading of the scenario) x Parameter (Critical Region or Syllables coefficient) ANOVA, with the last three variables repeated measures. An Expertise x Parameter interaction, F(1,73)=6.6, p < .05, showed that experts devoted more time to reading critical versus noncritical information, and less time per syllable compared to the novices. This finding was moderated by complexity and trial, F(1,73)=7.7, p < .01, such that expert allocation to critical regions was greater when reading the complex scenarios for the first time. Age did not moderate these effects.

Expert (E=92%, N=88% correct), F(1,73) =6.5, p<.05, and younger pilots (Y=93%, O=87%), F(1,73) =10.7, p<.01, answered the scenario questions more accurately. Allocation of reading time to critical regions was positively related to comprehension of complex scenarios, r=.23, p < .05, which in turn was associated with accurately identifying, r=.21, p < .08, and elaborating on, r=.32, p < .01, solutions to problems in these scenarios.

DISCUSSION

Older expert pilots were at least as accurate as younger experts in identifying the problems in the scenarios and solutions to these problems. Novices identified appropriate solutions to the problems less often and described the problems and solutions less elaborately than experts. These findings are consistent with previous studies showing that experts develop more differentiated problem representations and engage in more elaborate situation assessment to support decision making (Wickens & Hollands, 2000).

Perhaps most important, we found that novice pilots showed age-related declines in decision making (identifying appropriate solutions and elaborating problem descriptions), while experts tended to show age-related increases. The reading time findings suggest experts may have maintained decision-making accuracy by allocating more time to reading critical information for more complex scenarios, while novices spent more time reading longer words, suggesting an expertise advantage in situation model but not text-base level processing. This finding is consistent with the general finding that experts focus more than novices do on the most relevant aspects of problems, reflecting a more elaborate problem representation (O’Hare, 2003). Our study suggests that experts are not only adept at identifying relevant cues in problem situations, but this strategy may help older experts to maintain performance in the face of cognitive declines.

The reading time allocation and comprehension findings may also have implications for training attentional strategies in support of pilots’ situation awareness and decision making.

ACKNOWLEDGMENTS

This material is based upon work supported by the National Institutes of Health under Award R01 AG13936. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the NIH.

REFERENCES

Adams, M. J., Tenney, Y. J., & Pew, R. W. (1995). Situation awareness and the cognitive management of complex systems. Human Factors, 37, 85-104.

Bellenkes, M. A., Wickens, C. D., & Kramer, A. F. (1997). Visual scanning and pilot expertise: The role of attentional flexibility and mental model development. Aviation, Space and Environmental Medicine, 68, 569-579.

Ericsson, K. A., & Charness, N. (1994). Expert performance: Its structure and acquisition. American Psychologist, 49, 725-747.

Ericsson, K. A., & Kintsch, W. (1995). Long-term working memory. Psychological Review, 102, 211-245.

Klein, G. A. (1993). A recognition-primed decision (RPD) model of rapid decision-making. In G. A. Klein & J. Orasanu & R. Calderwood & Zsambok (Eds.), Decision making in action: Models and methods (pp. 138-147). Norwood, NJ: Ablex.

Miller, L. M. S. (2003). The effects of age and domain knowledge on text processing. Journal of Gerontology: Psychological Sciences, 58B.

Mohler, S. R. (1981). Reasons for eliminating the "Age 60" regulation for airline pilots. Aviation, Space and Environmental Medicine, 52, 445-454.

Morrow, D. G., Ridolfo, H. E., Menard, W. E., Sanborn, A., Stine-Morrow, E. A. L., Magnor, C., Herman, L., Teller, T., & Bryant, D. (2003). Environmental support promotes expertise-based mitigation of age differences in pilot communication tasks. Psychology and Aging, 18, 268-284.

O'Hare, D. (2003). Aeronautical decision making: Metaphors, models, and methods. In P. S. Tsang & M. A. Vidulich (Eds.), Principles and practice of aviation psychology (pp. 201-238). Mahwah, NJ: Erlbaum.

Orasanu, J., & Fischer, U. (1997). Finding decisions in natural environments: The view from the cockpit. In C. Zsambok & G. A. Klein (Eds.), Naturalistic decision-making (pp. 343-357). Mahwah, NJ: Erlbaum.

Salthouse, T. A. (1991). Theoretical perspectives on cognitive aging. Hillsdale, NJ: Erlbaum.

Stine-Morrow, E. A. L., Gagne, D. D., Morrow, D. G., & DeWall, B. H. (2004). Age differences in rereading. Memory and Cognition, in press.

Stine-Morrow, E. A. L., Morrow, D. G., & Leno, R. (2002). Aging and resource allocation to the situation model in narrative understanding. Journal of Gerontology: Psychological Sciences, 57B, P291-P297.

Stokes, A., Belger, A., & Zhang, K. (1990). Investigation of factors comprising a model of pilot decision making: Part II. Anxiety and cognitive strategies in expert and novice aviators (ARL-90-8/SCEEE-90-2). Savoy, IL: University of Illinois, Aviation Research Laboratory.

Wickens, C. D., & Hollands, J. G. (2000). Engineering psychology and human performance (3rd ed.). Upper Saddle River, NJ: Prentice Hall.

Wickens, C. D., Stokes, A., Barnett, B., & Hyman, F. (1993). The effects of stress on pilot judgment in a MIDIS simulator. In O. Svenson & A. J. Maule (Eds.), Time pressure and stress in human judgment and decision-making (pp. 271-292). New York: Plenum Press.