Abstract
This paper addresses the performance of a user simulator in producing interactions with a spoken dialog system which include usability problems. Simple and general models of user behavior and speech understanding performance, as they are known in early design stages, are shown to detect up to 85% of the usability problems found in a real user test. Data can be reordered to allow a quick manual inspection of the test result. Thus, in an iterative design process small iterations, each incorporating knowledge gained from simulations, are recommended to efficiently improve the system design.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Ai, H. and Weng, F. (2008) User Simulation as Testing for Spoken Dialog Systems, in: Proc. of SIGdial 2008, Columbus, Ohio, pp. 164–171.
Araki, M. and Doshita, S. (1997) Automatic Evaluation Environment for Spoken Dialogue Systems, in: Proc. of ECAI ’96, Springer, London, UK, pp. 183–194.
Chung, G. (2004) Developing a flexible spoken dialog system using simulation, in: Proc. of ACL ’04, Barcelona, Spain.
Cohen, M. H., Giangola, J. P., Balogh, J. (2004) Voice User Interface Design, Addison-Wesley, Boston, USA.
Desurvire, H. W. (1994) Faster, Cheaper!! Are Usability Inspection Methods as Effective as Empirical Testing? In: J. Nielsen and R. L. Mack (eds), Usability Inspection Methods, John Wiley and Sons, New York, NY, USA.
Eckert, W., Levin, E., Pieraccini, R. (1997) User Modeling for Spoken Dialogue System Evaluation, in: Proc. of ASRU 1997, Santa Barbara, CA, USA.
Engelbrecht, K.-P., Quade, M., Möller, S. (2009) Analysis of a New Simulation Approach to Dialogue System Evaluation. Speech Communication, 51, pp. 1234–1252.
Ito, A., Shimada, K., Suzuki, M., Makino, S. (2006) A User Simulator Based on VoiceXML for Evaluation of Spoken Dialog Systems, in: Proc. of Interspeech 2006, Pittsburgh, PA, USA.
ITU-T Recommendation P.851 (2003) Subjective Quality Evaluation of Telephone Services Based on Spoken Dialogue Systems. International Telecommunication Union, Geneva, Switzerland.
ITU-T Supplement 24 to P-Series Recommendations (2005) Parameters Describing the Interaction with Spoken Dialogue Systems. International Telecommunication Union, Geneva, Switzerland.
Kieras, D. E. (2003) Model-based Evaluation. In: Jacko, J. and Sears, A. (eds), The Human-Computer Interaction Handbook, Erlbaum, Mahwah, NJ, USA, pp. 1191–1208.
López-Cózar, R., de la Torre, A., Segura, J. C., Rubio, A. J. (2003) Assessment of dialogue systems by means of a new simulation technique. Speech Communication, 40(3), pp. 387–407.
López-Cózar, R., Callejas, Z., McTear, M. (2006) Testing the Performance of Spoken Dialogue Systems by Means of an Artificially Simulated User. Artificial Intelligence Review 26, pp. 291–323.
López-Cózar, R., Espejo, G., Callejas, Z., Gutiérrez, A., Griol, D. (2009) Assessment of Spoken Dialogue Systems by Simulating Different Levels of User Cooperativeness, in: Proc. of the IWSDS 2009, Kloster Irsee, Germany.
Möller, S. (2005) Quality of Telephone-based Spoken Dialog Systems. Springer, New York, USA.
Nielsen, J. (1993) Usability Engineering, Academic Press, San Diego, CA.
Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H., Young, S. (2007) Agenda-based User Simulation for Boot-strapping a POMDP Dialogue System, in: Proc. of HLT/NAACL, Rochester, NY, USA.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer Science+Business Media, LLC
About this paper
Cite this paper
Engelbrecht, KP., Schmidt, S., Möller, S. (2011). Performance of an Ad-hoc User Simulation in a Formative Evaluation of a Spoken Dialog System. In: Delgado, RC., Kobayashi, T. (eds) Proceedings of the Paralinguistic Information and its Integration in Spoken Dialogue Systems Workshop. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-1335-6_27
Download citation
DOI: https://doi.org/10.1007/978-1-4614-1335-6_27
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-1334-9
Online ISBN: 978-1-4614-1335-6
eBook Packages: EngineeringEngineering (R0)