Abstract
Context: Task-oriented conversational systems demand a high volume of data to understand human language. One of the major challenges of Natural Language Processing (NLP) is the lack of structured annotated data to improve and refine language models, therefore, institutions often generate or mine their own data and have to annotate it themselves.
Problem: The annotation process is time-consuming and costly process that usually results in errors due to human fatigue and often acts as the blocking phase for many smaller teams developing AI. Companies frequently report scarcity and poor data quality when developing these systems.
Solution: This paper presents Assis, a modular, adaptable tool for semi-automatic annotation (manual and AI annotation). The tool automates and organizes the intentions and entities in task-oriented conversations. Our proposal combines components that facilitate the visual assimilation of the annotation process. Assis can be embedded with continuously refined language models based on previously annotated sentences.
IS theory: Assis was developed with the idea of Design Theory in mind, using its base of knowledge to evaluate the existing and proposed tools to its goal of facilitating annotation.
Method: Empirical results from user experience in real-life case studies and satisfaction with both the annotation results as well as the user experience, in comparison to the same study groups conducting the annotation without tools or in another software, using a feedback form after use.
Results: During one of the case studies, the tool was used to annotate more than 800 messages, with user feedback relating a high satisfaction with the reduction of the required time.
Contributions and Impact in the IS area: The tool innovates with its deployless architecture, modularity and adaptability, while introducing two new concepts for text annotation: dialogue topics and entity propagation.