Few-shot out-of-scope intent classification: analyzing the robustness of prompt-based learning

Abstract

Out-of-scope (OOS) intent classification is an emerging field in conversational AI research. The goal is to detect out-of-scope user intents that do not belong to a predefined intent ontology. However, establishing a reliable OOS detection system is challenging due to limited data availability. This situation necessitates solutions rooted in few-shot learning techniques. For such few-shot text classification tasks, prompt-based learning has been shown more effective than conventionally finetuned large language models with a classification layer on top. Thus, we advocate for exploring prompt-based approaches for OOS intent detection. Additionally, we propose a new evaluation metric, the Area Under the In-scope and Out-of-Scope Characteristic curve (AU-IOC). This metric addresses the shortcomings of current evaluation standards for OOS intent detection. AU-IOC provides a comprehensive assessment of a model’s dual performance capacities: in-scope classification accuracy and OOS recall. Under this new evaluation method, we compare our prompt-based OOS detector against 3 strong baseline models by exploiting the metadata of intent annotations, i.e., intent description. Our study found that our prompt-based model achieved the highest AU-IOC score across different data regimes. Further experiments showed that our detector is insensitive to a variety of intent descriptions. An intriguing finding shows that for extremely low data settings (1- or 5-shot), employing a naturally phrased prompt template boosts the detector’s performance compared to rather artificially structured template patterns.

Appendices

Appendix A IOC curves at 1-50 shots

Figures 8, 9, 10, 11, 12 and 13 plot IOC curves of different models in 1-50 shot settings for SNIPS, Facebook, CLINC-Banking, Stackoverflow, HWU64 and BANKING dataset respectively.

Fig. 9

IOC curves ($Acc_{in}$ vs. $R_{oos}$) of models in 1-50 shot settings evaluated on Facebook (test set). Better viewed in color

Fig. 10

IOC curves ($Acc_{in}$ vs. $R_{oos}$) of models in 1-50 shot settings evaluated on CLINC-Banking (test set). Better viewed in color

Fig. 11

IOC curves ($Acc_{in}$ vs. $R_{oos}$) of models in 1-50 shot settings evaluated on Stackoverflow (test set). Better viewed in color

Fig. 12

IOC curves ($Acc_{in}$ vs. $R_{oos}$) of models in 1-50 shot settings evaluated on HWU64 (test set). Better viewed in color

Fig. 13

IOC curves ($Acc_{in}$ vs. $R_\textit{oos}$) of models in 1-50 shot settings evaluated on BANKING (test set). Better viewed in color

Appendix B Confidence score distributions of the other 3 datasets at 5-shot

Figure 14 shows the confidence score distributions of the 4 architectures on 3 datasets (SNIPS, Facebook and HWU64) at 5-shot.

Fig. 14

Confidence score histogram at the 5-shot setting on the test set of (a-d) SNIPS, (e-h) Facebook and (i-l) HWU64. Best viewed in color

Appendix C Inference speed

Figure 15 illustrates the inference throughput against the number of in-scope classes (denoted as L). To ensure a fair comparison between the models and to aptly simulate the online evaluation setting, we standardized the input batch size to 1 across all models. This means that each batch contains only a single user question. We observed that the throughput of the Softmax model remains relatively stable (approximately 62 instances/s) irrespective of the variations in L. The Softmax model bypasses the one-vs-all binary classification, thereby exhibiting speed insensitivity. In the case of the Siamese model, we implemented a strategy to cache the intent label embedding to foster efficiency. However, despite this optimization, we found that the computational demand for the cosine similarity operation escalates as L increases. In contrast, the throughput for the other two models is much smaller when L increases over 14. Notably, the prompt-based model surpasses others in achieving higher AU-IOC scores, albeit at the expense of reduced inference throughput, particularly when L exceeds 10. A significant factor contributing to this reduced speed is the tensor extraction operations involved in the prompt-based model, requiring data transfer between the GPU and CPU, which is time-consuming. While our primary focus in this study remains on scrutinizing the robustness of different models in handling the Out-of-Scope (OOS) intent detection task, we acknowledge that optimizing the inference speed is a critical aspect that warrants attention in future work. It is also pertinent to note that the inference time is contingent upon the hardware utilized during the evaluation, implying that a change in hardware could potentially alter the throughput numbers reported.

Fig. 15

Inference throughput v.s. number of in-scope classes. All the throughput numbers are computed with a single NVIDIA GTX-1080Ti GPU