iBet uBet web content aggregator. Adding the entire web to your favor.

Link to original content: https://doi.org/10.1145/3450337.3483495

PathOS+: A New Realm in Expert Evaluation | Extended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play

Work in Progress

PathOS+: A New Realm in Expert Evaluation

Authors:

Stevie Cheryl Francesca Sansalone,

Pejman Mirza-BabaeiAuthors Info & Claims

CHI PLAY '21: Extended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play

Pages 122 - 127

https://doi.org/10.1145/3450337.3483495

Published: 15 October 2021 Publication History

Abstract

Expert evaluation is commonly employed in usability research as it is fast and cost-effective. However, as it heavily relies on evaluators’ expertise, it is associated with problems of subjective interpretation. This is particularly noticeable in the games user research (GUR) field as games often aim to reach broad audiences and complex experiences in comparison to other applications. One way to improve the problems associated with subjective interpretation is to augment the evaluation with data collected from players. However collecting data from players can be time-consuming, expensive, and challenging to construct representative player samples. Building on our previous research on automated agent-based playtesting, in this work-in-progress paper we detail our new project in supplementing expert evaluation with simulated player data generated by a configurable population of artificial intelligence (AI) players.

Supplementary Material

Presentation slides (p122-CHI_WIP_Presentation.pdf)

Download
819.90 KB

References

[1]

S. Agarwal, C. Herrmann, G. Wallner, and F. Beck. 2020. Visualizing AI Playtesting Data of 2D Side-scrolling Games. In 2020 IEEE Conference on Games (CoG). 572–575. https://doi.org/10.1109/CoG47356.2020.9231915

[2]

Peter Braun, Alfredo Cuzzocrea, Timothy Keding, Carson Leung, Adam Padzor, and Dell Sayson. 2017. Game Data Mining: Clustering and Visualization of Online Game Data in Cyber-Physical Worlds. Procedia Computer Science 112 (12 2017), 2259–2268. https://doi.org/10.1016/j.procs.2017.08.141

Digital Library

[3]

Judeth Oden Choi, Jodi Forlizzi, Michael Christel, Rachel Moeller, MacKenzie Bates, and Jessica Hammer. 2016. Playtesting with a Purpose. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (Austin, Texas, USA) (CHI PLAY ’16). Association for Computing Machinery, New York, NY, USA, 254–265. https://doi.org/10.1145/2967934.2968103

Digital Library

[4]

J. Clement. 2021. Number of games released on Steam 2021. https://www.statista.com/statistics/552623/number-games-released-steam/

[5]

Gilbert Cockton and Alan Woolrych. 2001. Understanding Inspection Methods: Lessons from an Assessment of Heuristic Evaluation. (01 2001). https://doi.org/10.1007/978-1-4471-0353-0_11

[6]

Edgar Bermudez Contreras, Benjamin J Clark, and A. Wilber. 2020. The Neuroscience of Spatial Navigation and the Relationship to Artificial Intelligence. Frontiers in Computational Neuroscience 14 (2020).

[7]

Fernando de Mesentier Silva, Igor Borovikov, John Kolen, Navid Aghdaie, and Kazi A. Zaman. 2018. Exploring Gameplay With AI Agents. CoRR abs/1811.06962(2018). arxiv:1811.06962http://arxiv.org/abs/1811.06962

[8]

Fernando de Mesentier Silva, Scott Lee, Julian Togelius, and Andy Nealen. 2017. AI-Based Playtesting of Contemporary Board Games. In Proceedings of the 12th International Conference on the Foundations of Digital Games(FDG ’17). Association for Computing Machinery, New York, NY, USA, Article 13, 10 pages. https://doi.org/10.1145/3102071.3102105

Digital Library

[9]

Heather Desurvire and Charlotte Wiberg. 2009. Game Usability Heuristics (PLAY) for Evaluating and Designing Better Games: The Next Iteration. 557–566. https://doi.org/10.1007/978-3-642-02774-1_60

Digital Library

[10]

Anders Drachen, Christian Thurau, Julian Togelius, Georgios N. Yannakakis, and Christian Bauckhage. 2013. Game Data Mining. Springer London, London, 205–253. https://doi.org/10.1007/978-1-4471-4769-5_12

[11]

Nathan Ensmenger. 2012. Is chess the drosophila of artificial intelligence? A social history of an algorithm. Social Studies of Science 42 (2012), 30 – 5.

[12]

Ben Gilbert. 2020. Video-game industry revenues grew so much during the pandemic that they reportedly exceeded sports and film combined. Business Insider (2020). https://www.businessinsider.com/video-game-industry-revenues-exceed-sports-and-film-combined-idc-2020-12

[13]

Ebba Thora Hvannberg, Effie Lai-Chong Law, and Marta Kristín Lárusdóttir. 2007. Heuristic evaluation: Comparing ways of finding and reporting usability problems. Interacting with Computers 19, 2 (2007), 225–240. https://doi.org/10.1016/j.intcom.2006.10.001 HCI Issues in Computer Games.

Digital Library

[14]

Effie Lai-Chong Law and Ebba Thora Hvannberg. 2004. Analysis of Strategies for Improving and Estimating the Effectiveness of Heuristic Evaluation. In Proceedings of the Third Nordic Conference on Human-Computer Interaction (Tampere, Finland) (NordiCHI ’04). Association for Computing Machinery, New York, NY, USA, 241–250. https://doi.org/10.1145/1028014.1028051

Digital Library

[15]

Jialin Liu, Sam Snodgrass, Ahmed Khalifa, Sebastian Risi, Georgios Yannakakis, and Julian Togelius. 2020. Deep Learning for Procedural Content Generation. Neural Computing and Applications (10 2020).

[16]

Pejman Mirza-Babaei. 2018. UX Research for the Gaming Industry Requires Looking through a Unique Lens. User Experience Magazine(2018).

[17]

Pejman Mirza-Babaei, Naeem Moosajee, and Brandon Drenikow. 2016. Playtesting for Indie Studios. In Proceedings of the 20th International Academic Mindtrek Conference (Tampere, Finland) (AcademicMindtrek ’16). Association for Computing Machinery, New York, NY, USA, 366–374. https://doi.org/10.1145/2994310.2994364

Digital Library

[18]

Pejman Mirza-Babaei, Veronica Zammitto, Jörg Niesenhaus, Mirweis Sangin, and Lennart Nacke. 2013. Games user research: practice, methods, and applications. 3219–3222. https://doi.org/10.1145/2468356.2479651

Digital Library

[19]

Volodymyr Mnih, K. Kavukcuoglu, D. Silver, Andrei A. Rusu, J. Veness, Marc G. Bellemare, A. Graves, Martin A. Riedmiller, A. Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, A. Sadik, Ioannis Antonoglou, Helen King, D. Kumaran, Daan Wierstra, S. Legg, and D. Hassabis. 2015. Human-level control through deep reinforcement learning. Nature 518(2015), 529–533.

[20]

Braulio Murillo, Jose Pow Sang, and Freddy Paz. 2018. Heuristic Evaluation and Usability Testing as Complementary Methods: A Case Study. In Design, User Experience, and Usability: Theory and Practice, Aaron Marcus and Wentao Wang (Eds.). Springer International Publishing, Cham, 470–478.

[21]

Lennart Nacke, J. Niesenhaus, S. Engl, Alessandro Canossa, Kai Kuikkaniemi, and Thomas Immich. 2010. Bringing digital games to user research and user experience. CEUR Workshop Proceedings 634 (01 2010).

[22]

Mark J. Nelson and Michael Mateas. 2007. Towards Automated Game Design. In AI*IA 2007: Artificial Intelligence and Human-Oriented Computing, Roberto Basili and Maria Teresa Pazienza (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 626–637.

Digital Library

[23]

Aida Nematzadeh, Sebastian Ruder, and Dani Yogatama. 2020. ON MEMORY IN HUMAN AND ARTIFICIAL LANGUAGE PROCESSING SYSTEMS.

[24]

Jakob Nielsen and Robert L. Mack. 1994. Usability Inspection Methods. Wiley, New York.

Digital Library

[25]

Jakob Nielsen and Rolf Molich. 1990. Heuristic Evaluation of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, USA) (CHI ’90). Association for Computing Machinery, New York, NY, USA, 249–256. https://doi.org/10.1145/97243.97281

Digital Library

[26]

J. Pfau, A. Liapis, G. Volkmar, G. N. Yannakakis, and R. Malaka. 2020. Dungeons Replicants: Automated Game Balancing via Deep Player Behavior Modeling. In 2020 IEEE Conference on Games (CoG). 431–438. https://doi.org/10.1109/CoG47356.2020.9231958

[27]

N. Shaker, M. Shaker, and J. Togelius. 2013. Ropossum: An Authoring Tool for Designing, Optimizing and Solving Cut the Rope Levels. In AIIDE.

[28]

Samantha . Stahlke, Atiya Nova, and Pejman Mirza-Babaei. 2019. Artificial Playfulness: A Tool for Automated Agent-Based Playtesting. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3313039

Digital Library

[29]

Samantha Stahlke, Atiya Nova, and Pejman Mirza-Babaei. 2020. Artificial Players in the Design Process: Developing an Automated Testing Tool for Game Level and World Design. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (Virtual Event, Canada) (CHI PLAY ’20). Association for Computing Machinery, New York, NY, USA, 267–280. https://doi.org/10.1145/3410404.3414249

Digital Library

[30]

Samantha N. Stahlke and Pejman Mirza-Babaei. 2018. Usertesting Without the User: Opportunities and Challenges of an AI-Driven Approach in Games User Research. Comput. Entertain. 16, 2, Article 9 (April 2018), 18 pages. https://doi.org/10.1145/3183568

Digital Library

[31]

Wei-Siong Tan, Dahai Liu, and R. Bishu. 2009. Web evaluation: heuristic evaluation vs. user testing.International Journal of Industrial Ergonomics 39 (2009), 621–627.

[32]

Tommy Thompson. 2017. Games by ANGELINA: The AI Game Designer. https://towardsdatascience.com/angelina-6d4a6a311a4

[33]

Rafael Veras and C. Collins. 2020. Discriminability Tests for Visualization Effectiveness and Scalability. IEEE Transactions on Visualization and Computer Graphics 26 (2020), 749–758.

[34]

Gareth R. White, Pejman Mirza-babaei, Graham McAllister, and Judith Good. 2011. Weak Inter-Rater Reliability in Heuristic Evaluation of Video Games. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI EA ’11). Association for Computing Machinery, New York, NY, USA, 1441–1446. https://doi.org/10.1145/1979742.1979788

Digital Library

[35]

Georgios N. Yannakakis, P. Spronck, Daniele Loiacono, and E. André. 2013. Player Modeling. In Artificial and Computational Intelligence in Games.

Cited By

Khaleque BCook MGow J(2024)Experiments in Motivating Exploratory AgentsProceedings of the 19th International Conference on the Foundations of Digital Games10.1145/3649921.3659850(1-8)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3649921.3659850
Nova ASansalone SRobinson RMirza-Babaei P(2022)Charting the Uncharted with GUR: How AI Playtesting Can Supplement Expert EvaluationProceedings of the 17th International Conference on the Foundations of Digital Games10.1145/3555858.3555880(1-12)Online publication date: 5-Sep-2022
https://dl.acm.org/doi/10.1145/3555858.3555880

Index Terms

PathOS+: A New Realm in Expert Evaluation
1. Information systems
2. Software and its engineering

Index terms have been assigned to the content through auto-classification.

Recommendations

Charting the Uncharted with GUR: How AI Playtesting Can Supplement Expert Evaluation
FDG '22: Proceedings of the 17th International Conference on the Foundations of Digital Games

Despite the advantages of using expert evaluation as a method within games user research (GUR) (i.e. provides stakeholders low cost, rapid feedback), it does not always accurately reflect the general player’s experience. Testing the game out with real ...
Artificial Playfulness: A Tool for Automated Agent-Based Playtesting
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

Usertesting is commonly employed in games user research (GUR) to understand the experience of players interacting with digital games. However, recruitment and testing with human users can be laborious and resource-intensive, particularly for independent ...
Artificial Players in the Design Process: Developing an Automated Testing Tool for Game Level and World Design
CHI PLAY '20: Proceedings of the Annual Symposium on Computer-Human Interaction in Play

Iterative user-centred design has become a standard approach for developing interactive products. This process relies on prototyping and usertesting as early as possible to deliver a positive user experience and ensure that final products align with ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI PLAY '21: Extended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play

October 2021

414 pages

ISBN:9781450383561

DOI:10.1145/3450337

Editors:
Günter Wallner
JKU Linz, Austria & TU Eindhoven, The Netherlands
,
Alexander Meschtscherjakov
Paris Lodron University of Salzburg, Austria
,
Max Birk
Eindhoven University of Technology, The Netherlands
,
Jo Iacovides
University of York, UK
,
Mitchell McEwan
Macquarie University, Australia

Copyright © 2021 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2021

Check for updates

Author Tags

Qualifiers

Work in progress
Research
Refereed limited

Funding Sources

NSERC CREATE SWaGUR Grant
NSERC Discovery Grant

Conference

CHI PLAY '21

Sponsor:

SIGCHI

CHI PLAY '21: The Annual Symposium on Computer-Human Interaction in Play

October 18 - 21, 2021

Virtual Event, Austria

Acceptance Rates

Overall Acceptance Rate 421 of 1,386 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
123
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)3

Reflects downloads up to 27 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Khaleque BCook MGow J(2024)Experiments in Motivating Exploratory AgentsProceedings of the 19th International Conference on the Foundations of Digital Games10.1145/3649921.3659850(1-8)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3649921.3659850
Nova ASansalone SRobinson RMirza-Babaei P(2022)Charting the Uncharted with GUR: How AI Playtesting Can Supplement Expert EvaluationProceedings of the 17th International Conference on the Foundations of Digital Games10.1145/3555858.3555880(1-12)Online publication date: 5-Sep-2022
https://dl.acm.org/doi/10.1145/3555858.3555880

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents