{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,3]],"date-time":"2024-09-03T21:42:30Z","timestamp":1725399750962},"reference-count":56,"publisher":"Wiley","issue":"2","license":[{"start":{"date-parts":[[2014,7,23]],"date-time":"2014-07-23T00:00:00Z","timestamp":1406073600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Cognitive Science"],"published-print":{"date-parts":[[2015,3]]},"abstract":"Abstract<\/jats:title>Research into expertise is relatively common in cognitive science concerning expertise existing across many domains. However, much less research has examined how experts within the same domain assess the performance of their peer experts. We report the results of a modified think\u2010aloud study conducted with 18 pilots (6 first officers, 6 captains, and 6 flight examiners). Pairs of same\u2010ranked pilots were asked to rate the performance of a captain flying in a critical pre\u2010recorded simulator scenario. Findings reveal (a) considerable variance within performance categories, (b) differences in the process used as evidence in support of a performance rating, (c) different numbers and types of facts (cues) identified, and (d) differences in how specific performance events affect choice of performance category and gravity of performance assessment. Such variance is consistent with low inter\u2010rater reliability. Because raters exhibited good, albeit imprecise, reasons and facts, a fuzzy mathematical model of performance rating was developed. The model provides good agreement with observed variations.<\/jats:p>","DOI":"10.1111\/cogs.12152","type":"journal-article","created":{"date-parts":[[2014,7,23]],"date-time":"2014-07-23T11:04:02Z","timestamp":1406113442000},"page":"405-433","update-policy":"http:\/\/dx.doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Peer Assessment of Aviation Performance: Inconsistent for Good Reasons"],"prefix":"10.1111","volume":"39","author":[{"given":"Wolff\u2010Michael","family":"Roth","sequence":"first","affiliation":[{"name":"Faculty of Education University of Victoria"},{"name":"Griffith Institute of Educational Research Griffith University"}]},{"given":"Timothy J.","family":"Mavin","sequence":"additional","affiliation":[{"name":"Griffith Institute of Educational Research Griffith University"}]}],"member":"311","published-online":{"date-parts":[[2014,7,23]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0925-7535(00)00045-X"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF03173570"},{"key":"e_1_2_8_4_1","volume-title":"Dialogical imagination","author":"Bakhtin M. M.","year":"1981"},{"key":"e_1_2_8_5_1","doi-asserted-by":"crossref","first-page":"36","DOI":"10.4000\/books.pressesmines.967","volume-title":"The fourth resilience engineering symposium","author":"Bergstr\u00f6m J.","year":"2011"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327108IJAP1203_4"},{"key":"e_1_2_8_7_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog0502_2"},{"key":"e_1_2_8_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11135\u2010013\u20109888\u20103"},{"key":"e_1_2_8_9_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1551-6709.2011.01196.x"},{"key":"e_1_2_8_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10111-003-0136-9"},{"key":"e_1_2_8_11_1","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/5657.001.0001"},{"key":"e_1_2_8_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/0165-0114(79)90002-2"},{"key":"e_1_2_8_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/0165-0114(80)90002-0"},{"key":"e_1_2_8_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0165-0114(83)80117-1"},{"key":"e_1_2_8_15_1","first-page":"97","article-title":"Development of the NOTECHS (nontechnical skills) system for assessing pilots\u2019 skills","volume":"3","author":"Flin R.","year":"2003","journal-title":"Human Factors and Aerospace Safety"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10459-010-9250-7"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10459-006-9043-1"},{"key":"e_1_2_8_18_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327108IJAP1203_3"},{"key":"e_1_2_8_19_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1551-6709.2009.01087.x"},{"key":"e_1_2_8_20_1","doi-asserted-by":"publisher","DOI":"10.1525\/jlin.1993.3.1.3"},{"key":"e_1_2_8_21_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog2801_7"},{"key":"e_1_2_8_22_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327108IJAP1203_7"},{"key":"e_1_2_8_23_1","doi-asserted-by":"publisher","DOI":"10.1086\/653625"},{"key":"e_1_2_8_24_1","doi-asserted-by":"publisher","DOI":"10.1080\/10400419.2012.730331"},{"key":"e_1_2_8_25_1","volume-title":"Cognition in the wild","author":"Hutchins E.","year":"1995"},{"key":"e_1_2_8_26_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog1903_1"},{"key":"e_1_2_8_27_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.anthro.26.1.109"},{"key":"e_1_2_8_28_1","volume-title":"Intuition at work","author":"Klein G.","year":"2003"},{"key":"e_1_2_8_29_1","volume-title":"Fuzzy sets and fuzzy logic: Theory and applications","author":"Klir G. J.","year":"1995"},{"key":"e_1_2_8_30_1","doi-asserted-by":"publisher","DOI":"10.1525\/jlin.1997.7.1.98"},{"key":"e_1_2_8_31_1","doi-asserted-by":"publisher","DOI":"10.1177\/0265532211406422"},{"key":"e_1_2_8_32_1","doi-asserted-by":"publisher","DOI":"10.1027\/2192-0923\/a000041"},{"key":"e_1_2_8_33_1","doi-asserted-by":"publisher","DOI":"10.7208\/chicago\/9780226514642.001.0001"},{"key":"e_1_2_8_34_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1551-6709.2012.01237.x"},{"key":"e_1_2_8_35_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327108IJAP1203_5"},{"key":"e_1_2_8_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ergon.2008.01.014"},{"key":"e_1_2_8_37_1","first-page":"334","article-title":"A fuzzy method on determining of job and personnel evaluation results, and matching them with suggested model","volume":"17","author":"\u00d6zdaban I.","year":"2010","journal-title":"International Journal of Industrial Engineering"},{"key":"e_1_2_8_38_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327108IJAP1004_1"},{"key":"e_1_2_8_39_1","doi-asserted-by":"publisher","DOI":"10.1207\/s1532690xci2104_3"},{"key":"e_1_2_8_40_1","doi-asserted-by":"publisher","DOI":"10.1027\/2192-0923\/a000045"},{"key":"e_1_2_8_41_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1551-6709.2012.01253.x"},{"key":"e_1_2_8_42_1","doi-asserted-by":"publisher","DOI":"10.3758\/PBR.16.2.225"},{"key":"e_1_2_8_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssci.2012.06.010"},{"key":"e_1_2_8_44_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog1702_4"},{"key":"e_1_2_8_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0377-2217(01)00113-8"},{"key":"e_1_2_8_46_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1540-4781.1992.tb02574.x"},{"key":"e_1_2_8_47_1","doi-asserted-by":"publisher","DOI":"10.1080\/028134398750003241"},{"key":"e_1_2_8_48_1","first-page":"85","article-title":"Improving scoring consistency of flight performance through inter\u2010rater reliability analyses","volume":"26","author":"Smith M. V.","year":"2008","journal-title":"Collegiate Aviation Review"},{"key":"e_1_2_8_49_1","doi-asserted-by":"publisher","DOI":"10.1002\/tea.20221"},{"key":"e_1_2_8_50_1","volume-title":"Human\u2010machine reconfigurations: Plans and situated actions","author":"Suchman L.","year":"2007"},{"key":"e_1_2_8_51_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0022790"},{"key":"e_1_2_8_52_1","doi-asserted-by":"publisher","DOI":"10.1080\/07370008.2011.607929"},{"key":"e_1_2_8_53_1","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.45.1.104.27233"},{"key":"e_1_2_8_54_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691611406923"},{"key":"e_1_2_8_55_1","doi-asserted-by":"publisher","DOI":"10.2307\/3035022"},{"key":"e_1_2_8_56_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog2203_3"},{"key":"e_1_2_8_57_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2044-8325.1994.tb00562.x"}],"container-title":["Cognitive Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1111%2Fcogs.12152","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/cogs.12152","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,3]],"date-time":"2023-10-03T11:21:46Z","timestamp":1696332106000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/cogs.12152"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,7,23]]},"references-count":56,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2015,3]]}},"alternative-id":["10.1111\/cogs.12152"],"URL":"https:\/\/doi.org\/10.1111\/cogs.12152","archive":["Portico"],"relation":{},"ISSN":["0364-0213","1551-6709"],"issn-type":[{"value":"0364-0213","type":"print"},{"value":"1551-6709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,7,23]]},"assertion":[{"value":"2013-05-02","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-12-19","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-07-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}
