Predicting Group Performance Using Process Data in a Collaborative Assessment

Abstract

Technology-based assessments that involve collaboration among students offer many sources of process data, although it remains unclear which aspects of these data are most meaningful for making inferences about students’ collaborative skills. Recent research has focused mainly on theory-based rubrics for qualitative coding of process data (e.g., text from chat dialogues, click-stream data), but many reliability and validity issues arise in the application of such rubrics. In this research, we take a more data-driven approach to the problem. Data were collected from 122 dyads who interacted over online chat to complete a twelfth-grade mathematics assessment. We focus on features of chat and click-stream that can be extracted automatically, including the extent to which chat dialogue contained content from assessment materials; chat-based cues of affective tone and mirroring; and temporal synchronization in task-related activities. Using a block-wise linear regression, we show that process features of chat and click-stream accounted for 30.5% of the variation in group performance, after controlling for group members math proficiency and the total number of words in the chat dialogue. The full model explained 61% of the variation in group performance. Implications for the design and scoring of collaborative assessments are discussed.

Appendices

Appendix 1

This “Appendix” derives the group performance measure $\varDelta$ presented in Sect. 2.2.1 from the SC-IRT model developed by Halpin and Bergner (2018). The SC-IRT model describes the probability of both members of a dyad providing a correct response to a collaborative-solved item, denoted $R_i$, for $i = 1, \dots , I$. The group probability is modeled as a function of the probability of either partner solving the item when working individually, $P_{ij}$, for $j = 1, 2$. The model additionally includes a decision parameter, w, that describes how the individual probabilities combine to produce the group result. The model can be written:

$$\begin{aligned} R_i&= w(P_{i1} + P_{i2} - 2 P_{i1}\,P_{i2}) + P_{i1}\,P_{i2}. \end{aligned}$$

To obtain the scoring rule used in the present analysis, we first sum both sides of the equation to obtain the model-implied total scores, and then re-arrange to solve for w:

$$\begin{aligned} \sum _i R_i&= \sum _i \left( w (P_{i1} + P_{i2} - 2 P_{i1} \, P_{i2}) + P_{i1}\,P_{i2} \right) \\ {\widehat{Y}}&= w( \widehat{X}_1 + \widehat{X}_2 - 2 \widehat{X}_{12}) + \widehat{X}_{12} \\ \rightarrow&\\ \nonumber w&= \frac{{\widehat{Y}} - \widehat{X}_{12}}{\widehat{X}_1 + \widehat{X}_2 - 2 \widehat{X}_{12}} \end{aligned}$$

Eq. 1 of the paper is obtained by replacing the model-implied total scores with the observed total scores on the mean-equated individual and collaborative test forms. Finally, we divided the numerator and denominator by I to change the total scores to proportion correct. For further details on the theoretical motivation and properties of the SC-IRT model, please see Halpin and Bergner (2018).

Appendix 2

See Table 5.

Table 5 Descriptive statistics of study variables