{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,3]],"date-time":"2024-08-03T09:50:31Z","timestamp":1722678631397},"reference-count":44,"publisher":"MIT Press","issue":"2","funder":[{"DOI":"10.13039\/100000169","name":"Division of Behavioral and Cognitive Sciences","doi-asserted-by":"publisher","award":["NSF2020844"],"id":[{"id":"10.13039\/100000169","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,2,1]]},"abstract":"Abstract<\/jats:title>\n In reinforcement learning (RL) experiments, participants learn to make rewarding choices in response to different stimuli; RL models use outcomes to estimate stimulus\u2013response values that change incrementally. RL models consider any response type indiscriminately, ranging from more concretely defined motor choices (pressing a key with the index finger), to more general choices that can be executed in a number of ways (selecting dinner at the restaurant). However, does the learning process vary as a function of the choice type? In Experiment 1, we show that it does: Participants were slower and less accurate in learning correct choices of a general format compared with learning more concrete motor actions. Using computational modeling, we show that two mechanisms contribute to this. First, there was evidence of irrelevant credit assignment: The values of motor actions interfered with the values of other choice dimensions, resulting in more incorrect choices when the correct response was not defined by a single motor action; second, information integration for relevant general choices was slower. In Experiment 2, we replicated and further extended the findings from Experiment 1 by showing that slowed learning was attributable to weaker working memory use, rather than slowed RL. In both experiments, we ruled out the explanation that the difference in performance between two condition types was driven by difficulty\/different levels of complexity. We conclude that defining a more abstract choice space used by multiple learning systems for credit assignment recruits executive resources, limiting how much such processes then contribute to fast learning.<\/jats:p>","DOI":"10.1162\/jocn_a_01947","type":"journal-article","created":{"date-parts":[[2022,12,6]],"date-time":"2022-12-06T19:58:17Z","timestamp":1670356697000},"page":"314-330","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":6,"title":["Choice Type Impacts Human Reinforcement Learning"],"prefix":"10.1162","volume":"35","author":[{"given":"Milena","family":"Rmus","sequence":"first","affiliation":[{"name":"University of California, Berkeley"}]},{"given":"Amy","family":"Zou","sequence":"additional","affiliation":[{"name":"University of California, Berkeley"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-3751-3662","authenticated-orcid":true,"given":"Anne G. E.","family":"Collins","sequence":"additional","affiliation":[{"name":"University of California, Berkeley"},{"name":"Helen Wills Neuroscience Institute, Berkeley, CA"}]}],"member":"281","published-online":{"date-parts":[[2023,2,1]]},"reference":[{"key":"2024020103435330000_bib1","doi-asserted-by":"publisher","first-page":"3965","DOI":"10.1093\/cercor\/bhx259","article-title":"Beyond reward prediction errors: Human striatum updates rule values during learning","volume":"28","author":"Ballard","year":"2018","journal-title":"Cerebral Cortex"},{"key":"2024020103435330000_bib2","doi-asserted-by":"publisher","first-page":"e1003387","DOI":"10.1371\/journal.pcbi.1003387","article-title":"Cortical and hippocampal correlates of deliberation during model-based decisions for rewards in humans","volume":"9","author":"Bornstein","year":"2013","journal-title":"PLoS Computational Biology"},{"key":"2024020103435330000_bib3","doi-asserted-by":"publisher","first-page":"15958","DOI":"10.1038\/ncomms15958","article-title":"Reminders of past choices bias decisions for reward in humans","volume":"8","author":"Bornstein","year":"2017","journal-title":"Nature Communications"},{"key":"2024020103435330000_bib4","doi-asserted-by":"publisher","first-page":"262","DOI":"10.1016\/j.cognition.2008.08.011","article-title":"Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective","volume":"113","author":"Botvinick","year":"2009","journal-title":"Cognition"},{"key":"2024020103435330000_bib5","doi-asserted-by":"publisher","first-page":"1422","DOI":"10.1162\/jocn_a_01238","article-title":"The tortoise and the hare: Interactions between reinforcement learning and working memory","volume":"30","author":"Collins","year":"2018","journal-title":"Journal of Cognitive Neuroscience"},{"key":"2024020103435330000_bib6","doi-asserted-by":"publisher","first-page":"13747","DOI":"10.1523\/JNEUROSCI.0989-14.2014","article-title":"Working memory contributions to reinforcement learning impairments in schizophrenia","volume":"34","author":"Collins","year":"2014","journal-title":"Journal of Neuroscience"},{"key":"2024020103435330000_bib7","doi-asserted-by":"publisher","first-page":"4332","DOI":"10.1523\/JNEUROSCI.2700-16.2017","article-title":"Working memory load strengthens reward prediction errors","volume":"37","author":"Collins","year":"2017","journal-title":"Journal of Neuroscience"},{"key":"2024020103435330000_bib8","doi-asserted-by":"publisher","first-page":"1024","DOI":"10.1111\/j.1460-9568.2011.07980.x","article-title":"How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis","volume":"35","author":"Collins","year":"2012","journal-title":"European Journal of Neuroscience"},{"key":"2024020103435330000_bib9","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1037\/a0030852","article-title":"Cognitive control over learning: Creating, clustering, and generalizing task-set structure","volume":"120","author":"Collins","year":"2013","journal-title":"Psychological Review"},{"key":"2024020103435330000_bib10","doi-asserted-by":"publisher","first-page":"2502","DOI":"10.1073\/pnas.1720963115","article-title":"Within- and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory","volume":"115","author":"Collins","year":"2018","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2024020103435330000_bib11","doi-asserted-by":"publisher","first-page":"1204","DOI":"10.1016\/j.neuron.2011.02.027","article-title":"Model-based influences on humans' choices and striatal prediction errors","volume":"69","author":"Daw","year":"2011","journal-title":"Neuron"},{"key":"2024020103435330000_bib12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3758\/s13428-014-0458-y","article-title":"jsPsych: A JavaScript library for creating behavioral experiments in a web browser","volume":"47","author":"De Leeuw","year":"2015","journal-title":"Behavior Research Methods"},{"key":"2024020103435330000_bib13","doi-asserted-by":"publisher","first-page":"29381","DOI":"10.1073\/pnas.1912330117","article-title":"Computational evidence for hierarchically structured reinforcement learning in humans","volume":"117","author":"Eckstein","year":"2020","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2024020103435330000_bib14","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1016\/j.cognition.2019.01.009","article-title":"How the inference of hierarchical rules unfolds over time","volume":"185","author":"Eckstein","year":"2019","journal-title":"Cognition"},{"key":"2024020103435330000_bib15","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1016\/j.cobeha.2021.06.004","article-title":"What do reinforcement learning models measure? Interpreting model parameters in cognition and neuroscience","volume":"41","author":"Eckstein","year":"2021","journal-title":"Current Opinion in Behavioral Sciences"},{"key":"2024020103435330000_bib16","doi-asserted-by":"publisher","first-page":"1768","DOI":"10.1038\/s41467-017-01874-w","article-title":"Feature-based learning improves adaptability without compromising precision","volume":"8","author":"Farashahi","year":"2017","journal-title":"Nature Communications"},{"key":"2024020103435330000_bib17","doi-asserted-by":"publisher","first-page":"13157","DOI":"10.1523\/JNEUROSCI.2701-11.2011","article-title":"Feedback timing modulates brain systems for learning in humans","volume":"31","author":"Foerde","year":"2011","journal-title":"Journal of Neuroscience"},{"key":"2024020103435330000_bib18","doi-asserted-by":"publisher","first-page":"16311","DOI":"10.1073\/pnas.0706111104","article-title":"Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning","volume":"104","author":"Frank","year":"2007","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2024020103435330000_bib19","doi-asserted-by":"publisher","first-page":"1320","DOI":"10.3758\/s13423-014-0790-3","article-title":"Do learning rates adapt to the distribution of rewards?","volume":"22","author":"Gershman","year":"2015","journal-title":"Psychonomic Bulletin & Review"},{"key":"2024020103435330000_bib20","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1016\/j.cub.2009.01.063","article-title":"Attention alters visual plasticity during exposure-based learning","volume":"19","author":"Gutnisky","year":"2009","journal-title":"Current Biology"},{"key":"2024020103435330000_bib21","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1016\/j.jmp.2018.09.002","article-title":"The statistical structures of reinforcement learning with asymmetric value updates","volume":"87","author":"Katahira","year":"2018","journal-title":"Journal of Mathematical Psychology"},{"key":"2024020103435330000_bib22","doi-asserted-by":"publisher","first-page":"1864","DOI":"10.1523\/JNEUROSCI.4920-12.2013","article-title":"Choice coding in frontal cortex during stimulus-guided or action-guided decision-making","volume":"33","author":"Luk","year":"2013","journal-title":"Journal of Neuroscience"},{"key":"2024020103435330000_bib23","doi-asserted-by":"publisher","first-page":"100732","DOI":"10.1016\/j.dcn.2019.100732","article-title":"Disentangling the systems contributing to changes in learning during adolescence","volume":"41","author":"Master","year":"2020","journal-title":"Developmental Cognitive Neuroscience"},{"key":"2024020103435330000_bib24","doi-asserted-by":"publisher","first-page":"6797","DOI":"10.1073\/pnas.1523669113","article-title":"Credit assignment in movement-dependent reinforcement learning","volume":"113","author":"McDougle","year":"2016","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2024020103435330000_bib25","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/j.cobeha.2016.04.003","article-title":"Taming the beast: Extracting generalizable knowledge from computational models of cognition","volume":"11","author":"Nassar","year":"2016","journal-title":"Current Opinion in Behavioral Sciences"},{"key":"2024020103435330000_bib26","doi-asserted-by":"publisher","first-page":"1544","DOI":"10.1038\/s41593-019-0470-8","article-title":"Learning task-state representations","volume":"22","author":"Niv","year":"2019","journal-title":"Nature Neuroscience"},{"key":"2024020103435330000_bib27","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1523\/JNEUROSCI.5498-10.2012","article-title":"Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain","volume":"32","author":"Niv","year":"2012","journal-title":"Journal of Neuroscience"},{"key":"2024020103435330000_bib28","doi-asserted-by":"publisher","first-page":"546","DOI":"10.1038\/35107080","article-title":"Interactive memory systems in the human brain","volume":"414","author":"Poldrack","year":"2001","journal-title":"Nature"},{"key":"2024020103435330000_bib29","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1037\/h0024475","article-title":"Two-process learning theory: Relationships between Pavlovian conditioning and instrumental learning","volume":"74","author":"Rescorla","year":"1967","journal-title":"Psychological Review"},{"key":"2024020103435330000_bib30","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1016\/j.cobeha.2020.10.003","article-title":"The role of executive function in shaping reinforcement learning","volume":"38","author":"Rmus","year":"2021","journal-title":"Current Opinion in Behavioral Sciences"},{"key":"2024020103435330000_bib31","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1016\/j.ijchp.2019.07.006","article-title":"Cognitive flexibility and response inhibition in patients with obsessive-compulsive disorder and generalized anxiety disorder","volume":"20","author":"Rosa-Alc\u00e1zar","year":"2020","journal-title":"International Journal of Clinical and Health Psychology"},{"key":"2024020103435330000_bib32","doi-asserted-by":"publisher","first-page":"6902","DOI":"10.1523\/JNEUROSCI.0631-17.2017","article-title":"Effects of ventral striatum lesions on stimulus-based versus action-based reinforcement learning","volume":"37","author":"Rothenhoefer","year":"2017","journal-title":"Journal of Neuroscience"},{"key":"2024020103435330000_bib33","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1038\/nrn2737","article-title":"Advances in visual perceptual learning and plasticity","volume":"11","author":"Sasaki","year":"2010","journal-title":"Nature Reviews Neuroscience"},{"key":"2024020103435330000_bib34","doi-asserted-by":"publisher","first-page":"15871","DOI":"10.1073\/pnas.1821647116","article-title":"Credit assignment to state-independent task representations and its relationship with model-based decision making","volume":"116","author":"Shahar","year":"2019","journal-title":"Proceedings of the National Academy of Sciences, U.S.A."},{"key":"2024020103435330000_bib35","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1007\/3-540-45622-8_16","article-title":"Learning options in reinforcement learning","volume-title":"International symposium on abstraction, reformulation, and approximation","author":"Stolle","year":"2002"},{"key":"2024020103435330000_bib36","article-title":"Reinforcement learning: An introduction","volume-title":"Adaptive computation and machine learning","author":"Sutton","year":"2018","edition":"2nd ed."},{"key":"2024020103435330000_bib37","doi-asserted-by":"publisher","first-page":"1281","DOI":"10.1038\/nn.3188","article-title":"Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value","volume":"15","author":"Tai","year":"2012","journal-title":"Nature Neuroscience"},{"key":"2024020103435330000_bib38","article-title":"Learning to use working memory in partially observable environments through dopaminergic reinforcement","volume-title":"Advances in neural information processing systems","author":"Todd","year":"2008"},{"key":"2024020103435330000_bib39","doi-asserted-by":"publisher","first-page":"683","DOI":"10.1016\/j.neuron.2019.02.014","article-title":"Hippocampal contributions to model-based planning and spatial memory","volume":"102","author":"Vikbladh","year":"2019","journal-title":"Neuron"},{"key":"2024020103435330000_bib40","doi-asserted-by":"publisher","first-page":"192","DOI":"10.3758\/BF03206482","article-title":"AIC model selection using Akaike weights","volume":"11","author":"Wagenmakers","year":"2004","journal-title":"Psychonomic Bulletin & Review"},{"key":"2024020103435330000_bib41","doi-asserted-by":"publisher","first-page":"e49547","DOI":"10.7554\/eLife.49547","article-title":"Ten simple rules for the computational modeling of behavioral data","volume":"8","author":"Wilson","year":"2019","journal-title":"eLife"},{"key":"2024020103435330000_bib42","doi-asserted-by":"publisher","first-page":"270","DOI":"10.1126\/science.1223252","article-title":"Preference by association: How memory mechanisms in the hippocampus bias decisions","volume":"338","author":"Wimmer","year":"2012","journal-title":"Science"},{"key":"2024020103435330000_bib43","doi-asserted-by":"publisher","first-page":"643","DOI":"10.1037\/rev0000295","article-title":"Temporal and state abstractions for efficient learning, transfer, and composition in humans","volume":"128","author":"Xia","year":"2021","journal-title":"Psychological Review"},{"key":"2024020103435330000_bib44","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1162\/jocn_a_01808","article-title":"How working memory and reinforcement learning are intertwined: A cognitive, neural, and computational perspective","volume":"34","author":"Yoo","year":"2022","journal-title":"Journal of Cognitive Neuroscience"}],"container-title":["Journal of Cognitive Neuroscience"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/jocn\/article-pdf\/35\/2\/314\/2065862\/jocn_a_01947.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/jocn\/article-pdf\/35\/2\/314\/2065862\/jocn_a_01947.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,1]],"date-time":"2024-02-01T03:44:03Z","timestamp":1706759043000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/jocn\/article\/35\/2\/314\/114116\/Choice-Type-Impacts-Human-Reinforcement-Learning"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":44,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,2,1]]},"published-print":{"date-parts":[[2023,2,1]]}},"URL":"https:\/\/doi.org\/10.1162\/jocn_a_01947","relation":{},"ISSN":["0898-929X","1530-8898"],"issn-type":[{"value":"0898-929X","type":"print"},{"value":"1530-8898","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}
