iBet uBet web content aggregator. Adding the entire web to your favor.

Link to original content: https://doi.org/10.1145/3472749.3474784

An Aligned Rank Transform Procedure for Multifactor Contrast Tests | The 34th Annual ACM Symposium on User Interface Software and Technology

research-article

An Aligned Rank Transform Procedure for Multifactor Contrast Tests

Authors:

James J. Higgins,

Jacob O. WobbrockAuthors Info & Claims

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

Pages 754 - 768

https://doi.org/10.1145/3472749.3474784

Published: 12 October 2021 Publication History

Abstract

Data from multifactor HCI experiments often violates the assumptions of parametric tests (i.e., nonconforming data). The Aligned Rank Transform (ART) has become a popular nonparametric analysis in HCI that can find main and interaction effects in nonconforming data, but leads to incorrect results when used to conduct post hoc contrast tests. We created a new algorithm called ART-C for conducting contrast tests within the ART paradigm and validated it on 72,000 synthetic data sets. Our results indicate that ART-C does not inflate Type I error rates, unlike contrasts based on ART, and that ART-C has more statistical power than a t-test, Mann-Whitney U test, Wilcoxon signed-rank test, and ART. We also extended an open-source tool called ARTool with our ART-C algorithm for both Windows and R. Our validation had some limitations (e.g., only six distribution types, no mixed factorial designs, no random slopes), and data drawn from Cauchy distributions should not be analyzed with ART-C.

References

[1]

Marisela Abundis. 2001. Multiple comparison procedures in factorial designs using the aligned rank transformation. Thesis. Texas Tech University. https://ttu-ir.tdl.org/handle/2346/22569

[2]

Saleema Amershi, James Fogarty, and Daniel Weld. 2012. Regroup: interactive machine learning for on-demand group creation in social networks. In Proceedings of the 2012 CHI Conference on Human Factors in Computing Systems - CHI ’12. ACM, Austin, Texas, USA, 21–30. https://doi.org/10.1145/2207676.2207680

Digital Library

[3]

Shiri Azenkot, Kyle Rector, Richard Ladner, and Jacob Wobbrock. 2012. PassChords: secure multi-touch authentication for blind people. In Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility - ASSETS ’12. ACM, Boulder, Colorado, USA, 159–166. https://doi.org/10.1145/2384916.2384945

Digital Library

[4]

Eric Barefield and H. Mansouri. 2001. An empirical study of nonparametric multiple comparison procedures in randomized blocks. Journal of Nonparametric Statistics 13, 4 (2001), 591–604. https://doi.org/10.1080/10485250108832867

[5]

R. Clifford Blair and James J. Higgins. 1980. A Comparison of the Power of Wilcoxon’s Rank-Sum Statistic to that of Student’s t Statistic under Various Nonnormal Distributions. Journal of Educational Statistics 5, 4 (1980), 309–335. https://doi.org/10.2307/1164905

[6]

Stef van Buuren and Miranda Fredriks. 2001. Worm plot: a simple diagnostic device for modelling growth reference curves. Statistics in medicine 20, 8 (2001), 1259–1277. https://doi.org/10.1002/sim.746

[7]

D. Ciavardelli, C. Rossi, D. Barcaroli, S. Volpe, A. Consalvo, M. Zucchelli, A. De Cola, E. Scavo, R. Carollo, D. D’Agostino, F. Forlì, S. D’Aguanno, M. Todaro, G. Stassi, C. Di Ilio, V. De Laurenzi, and A. Urbani. 2014. Breast cancer stem cells rely on fermentative glycolysis and are sensitive to 2-deoxyglucose treatment. Cell Death & Disease 5, 7 (2014), e1336–e1336. https://doi.org/10.1038/cddis.2014.285

[8]

W. J. Conover and Ronald L. Iman. 1981. Rank Transformations as a Bridge Between Parametric and Nonparametric Statistics. The American Statistician 35, 3 (1981), 124–129. http://www.jstor.org/stable/2683975

[9]

R. F. Fawcett and K. C. Salter. 1984. A Monte Carlo study of the F test and three tests based on ranks of treatment effects in randomized block designs. Communications in Statistics: Simulation and Computation 13, 2(1984), 213–225. https://doi.org/10.1080/03610918408812368

[10]

Kara L. Feilich and George V. Lauder. 2015. Passive mechanical models of fish caudal fins: effects of shape and stiffness on self-propulsion. Bioinspiration & Biomimetics 10, 3 (2015), 036002. https://doi.org/10.1088/1748-3190/10/3/036002

[11]

Brigitte N Frederick. 1999. Fixed-, Random-, and Mixed-Effects ANOVA Models: A User-Friendly Guide for Increasing the Generalizability of ANOVA Results.(1999). https://eric.ed.gov/?id=ED426098

[12]

Milton Friedman. 1937. The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance. J. Amer. Statist. Assoc. 32, 200 (1937), 675–701. https://doi.org/10.2307/2279372

[13]

António Gaspar, André P Lourenço, Miguel Álvares Pereira, Pedro Azevedo, Roberto Roncon-Albuquerque, Jorge Marques, and Adelino F Leite-Moreira. 2018. Randomized controlled trial of remote ischaemic conditioning in ST-elevation myocardial infarction as adjuvant to primary angioplasty (RIC-STEMI). Basic research in cardiology 113, 3 (2018), 1–10. https://doi.org/10.1007/s00395-018-0672-3

[14]

Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, and Enrico Rukzio. 2017. ShareVR: Enabling Co-Located Experiences for Virtual Reality between HMD and Non-HMD Users. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17. ACM, Denver, Colorado, USA, 4021–4033. https://doi.org/10.1145/3025453.3025683

Digital Library

[15]

Nur Al-huda Hamdan, Adrian Wagner, Simon Voelker, Jürgen Steimle, and Jan Borchers. 2019. Springlets: Expressive, Flexible and Silent On-Skin Tactile Interfaces. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM, Glasgow, Scotland, UK, 1–14. https://doi.org/10.1145/3290605.3300718

Digital Library

[16]

James J. Higgins, R. Clifford Blair, and Suleiman Tashtoush. 1990. The Aligned Rank Transform Procedure. Conference on Applied Statistics in Agriculture (1990), 185–195. https://doi.org/10.4148/2475-7772.1443

[17]

James J. Higgins and Suleiman Tashtoush. 1994. An aligned rank transform test for interaction. Nonlinear World 1, 2 (1994), 201–211.

[18]

David C Hoaglin. 2011. Using quantiles to study shape. Exploring data tables, trends, and shapes 101 (2011), 417–460.

[19]

J. L. Hodges and E. L. Lehmann. 1962. Rank Methods for Combination of Independent Experiments in Analysis of Variance. The Annals of Mathematical Statistics 33, 2 (1962), 482–497. https://doi.org/10.1214/aoms/1177704575

[20]

Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics 6, 2 (1979), 65–70. https://www.jstor.org/stable/4615733

[21]

Eunice Jun, Maureen Daum, Jared Roesch, Sarah Chasins, Emery Berger, Rene Just, and Katharina Reinecke. 2019. Tea: A High-level Language and Runtime System for Automating Statistical Analysis. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology - UIST ’19. ACM, New Orleans, LA, USA, 591–603. https://doi.org/10.1145/3332165.3347940

Digital Library

[22]

Shaun K. Kane, Meredith Ringel Morris, Annuska Z. Perkins, Daniel Wigdor, Richard E. Ladner, and Jacob O. Wobbrock. 2011. Access overlays: improving non-visual access to large touch screens for blind users. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology - UIST ’11. ACM, Santa Barbara, California, USA, 273–282. https://doi.org/10.1145/2047196.2047232

Digital Library

[23]

Maurits Clemens Kaptein, Clifford Nass, and Panos Markopoulos. 2010. Powerful and consistent analysis of likert-type ratingscales. In Proceedings of the 2010 CHI Conference on Human Factors in Computing Systems - CHI ’10. ACM, Atlanta, Georgia, USA, 2391–2394. https://doi.org/10.1145/1753326.1753686

Digital Library

[24]

Matthew Kay. 2020. Contrast tests with ART. https://cran.r-project.org/web/packages/ARTool/vignettes/art-contrasts.html

[25]

Matthew Kay, Gregory L. Nelson, and Eric B. Hekler. 2016. Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16. ACM, San Jose, California, USA, 4521–4532. https://doi.org/10.1145/2858036.2858465

Digital Library

[26]

Kalimuthu Krishnamoorthy. 2006. Handbook of statistical distributions with applications. CRC Press. https://doi.org/10.1201/9781420011371

[27]

Dong Li. 2017. Robustness And Power Of The Student T, Welch-Aspin, Yuen, Tukey Quick, And Haga Tests. Thesis. Wayne State University. https://digitalcommons.wayne.edu/oa_dissertations/1722

[28]

H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics 18, 1 (1947), 50–60. https://www.jstor.org/stable/2236101

[29]

H. Mansouri. 1999. Multifactor analysis of variance based on the aligned rank transform technique. Computational Statistics & Data Analysis 29, 2 (1999), 177–189. https://doi.org/10.1016/S0167-9473(98)00077-2

Digital Library

[30]

H. Mansouri. 2015. Simultaneous inference based on rank statistics in linear models. Journal of Statistical Computation and Simulation 85, 4(2015), 660–674. https://doi.org/10.1080/00949655.2013.836292

[31]

H. Mansouri, R. L. Paige, and J. G. Surles. 2004. Aligned Rank Transform Techniques for Analysis of Variance and Multiple Comparisons. Communications in Statistics - Theory and Methods 33, 9 (2004), 2217–2232. https://doi.org/10.1081/STA-200026599

[32]

H. R. Neave and C. W. J. Granger. 1968. A Monte Carlo Study Comparing Various Two-Sample Tests for Differences in Mean. Technometrics 10, 3 (1968), 509–522. https://doi.org/10.2307/1267105

[33]

Daryle Alan Olson. 2013. The Efficacy Of Select Nonparametric And Distribution-Free Research Methods: Examining The Case Of Concomitant Heteroscedasticity And Effect Of Treatment. Thesis. Wayne State University. https://digitalcommons.wayne.edu/oa_dissertations/684

[34]

Kathleen Peterson. 2002. Six Modifications Of The Aligned Rank Transform Test For Interaction. Journal of Modern Applied Statistical Methods 1, 1 (2002), 100–109. https://doi.org/10.22237/jmasm/1020255240

[35]

M. L. Puri and P. K. Sen. 1969. A class of rank order tests for a general linear model. Annals of Mathematical Statistics 40, 4 (1969), 1325–1343. https://www.jstor.org/stable/2239597

[36]

Marta Revilla-León, Peng Jiang, Mehrad Sadeghpour, Wenceslao Piedra-Cascón, Amirali Zandinejad, Mutlu Özcan, and Vinayak R. Krishnamurthy. 2019. Intraoral digital scans—Part 1: Influence of ambient scanning light conditions on the accuracy (trueness and precision) of different intraoral scanners. The Journal of Prosthetic Dentistry 124, 3 (2019), 372–378. https://doi.org/10.1016/j.prosdent.2019.06.003

[37]

Scott J. Richter. 1999. Nearly exact tests in factorial experiments using the aligned rank transform. Journal of Applied Statistics 26, 2 (1999), 203–217. https://doi.org/10.1080/02664769922548

[38]

Judy Robertson and Maurits Kaptein. 2016. Modern Statistical Methods for HCI. Springer International Publishing. https://doi.org/10.1007/978-3-319-26633-6

Digital Library

[39]

Joan Sol Roo and Martin Hachet. 2017. One Reality: Augmenting How the Physical World is Experienced by combining Multiple Mixed Reality Modalities. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST ’17. ACM, Québec City, Québec, Canada, 787–795. https://doi.org/10.1145/3126594.3126638

Digital Library

[40]

K.C. Salter and R.F. Fawcett. 1985. A robust and powerful rank test of treatment effects in balanced incomplete block designs. Communications in Statistics: Simulation and Computation 14, 4(1985), 807–828. https://doi.org/10.1080/03610918508812475

[41]

K. C. Salter and R. F. Fawcett. 1993. The ART test of interaction: a robust and powerful rank test of interaction in factorial models. Communications in Statistics: Simulation and Computation 22, 1(1993), 137–153. https://doi.org/10.1080/03610919308813085

[42]

Shlomo S. Sawilowsky. 1990. Nonparametric tests of interaction in experimental design. Review of Educational Research 60, 1 (1990), 91–126. https://doi.org/10.2307/1170226

[43]

Shlomo S. Sawilowsky and R. Clifford Blair. 1992. A more realistic look at the robustness and Type II error properties of the t test to departures from population normality. Psychological Bulletin 111, 2 (1992), 352–360. https://doi.org/10.1037/0033-2909.111.2.352

[44]

Student. 1908. The Probable Error of a Mean. Biometrika 6, 1 (1908), 1–25. https://doi.org/10.2307/2331554

[45]

John W Tukey. 1949. Comparing individual means in the analysis of variance. Biometrics 5, 2 (1949), 99–114. https://doi.org/10.2307/3001913

[46]

James H. Ware. 1985. Linear Models for the Analysis of Longitudinal Studies. The American Statistician 39, 2 (1985), 95–101. https://doi.org/10.2307/2682803

[47]

Eric W Weisstein. 2004. Bonferroni correction. MathWorld–A Wolfram Web Resource(2004). https://mathworld.wolfram.com/BonferroniCorrection.html

[48]

Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods. Biometrics Bulletin 1, 6 (1945), 80–83. https://doi.org/10.2307/3001968

[49]

Leland Wilkinson, The Task Force on Statistical Inference, American Psychological Association, and Science Directorate. 1999. Statistical methods in psychology journals: Guidelines and explanations. American psychologist 54, 8 (1999), 594–604. https://doi.org/10.1037/0003-066X.54.8.594

[50]

Jacob O. Wobbrock. 2017. The relevance of nonparametric and semi-parametric statistics to HCI. In Workshop on ”Moving Transparent Statistics Forward.” 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17. ACM, Denver, Colorado, USA, Paper No. 2. http://faculty.washington.edu/wobbrock/pubs/chi-17.06.pdf

[51]

Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the 2011 CHI Conference on Human Factors in Computing Systems - CHI ’11. ACM, Vancouver, British Columbia, Canada, 143–146. https://doi.org/10.1145/1978942.1978963

Digital Library

[52]

Jacob O. Wobbrock and M. Kay. 2016. Nonparametric statistics in human-computer interaction. In Modern Statistical Methods for HCI, M. C. Robertson, J. & Kaptein (Ed.). Springer, Switzerland, Chapter 7, 135–170. https://doi.org/10.1007/978-3-319-26633-6_7

Cited By

Oo ZKAWAI NMATSUMIYA NHAYASHIDA M(2024)Japanese Teachers’ Attitudes and Self-Efficacy Toward Inclusive Education: Three-Way ANOVA and Regression AnalysesJournal of Education for Sustainability and Diversity10.57142/jesd.v3i1.6123:1(182-210)Online publication date: 29-Jul-2024
https://doi.org/10.57142/jesd.v3i1.612
Czeglédi ISpecziár APreiszner BBoros GBánó BMozsár ATakács PErős T(2024)Stable isotope analysis reveals diet niche partitioning between native species and the invasive black bullhead (Ameiurus melas Rafinesque, 1820)NeoBiota10.3897/neobiota.94.12249694(57-77)Online publication date: 26-Jul-2024
https://doi.org/10.3897/neobiota.94.122496
Miura TOkochi NMatsuo MYabu KKatagiri AYasu KSakajiri MIfukube T(2024)Examining How Interaural Differences Owing to Head Rotation during Walking Improve the Distance of Auditory Obstacle Perceptions among Individuals with Visual Impairment: A Case Study in Small-Scale Blind GroupLife10.3390/life1403035614:3(356)Online publication date: 7-Mar-2024
https://doi.org/10.3390/life14030356
Show More Cited By

Recommendations

Rank tests of association for exchangeable paired data

We describe two rank tests of association for paired exchangeable data motivated by the study of lifespans in twins. The pooled sample is ranked. The nonparametric test of association is based on R⁺, the sum of the smaller within-pair ranks. A second ...
Wald-type rank tests: A GEE approach

Factorial designs have been widely used in many scientific fields. Traditionally, such designs can be analyzed by the generalized linear mixed models (GLMMs). When making inference for the fixed effects in GLMM, however, even the robust generalized ...
Rank methods
Abstract
One of the major goals of statistics is to use data collected from a population to make inferences (hypothesis tests, confidence regions, and point estimators) about unknown attributes of the population. When there are good reasons to believe that ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

October 2021

1357 pages

ISBN:9781450386357

DOI:10.1145/3472749

Editors:
Jeffrey Nichols
Apple, USA
,
Ranjitha Kumar
UIUC, USA
,
Michael Nebeling
University of Michigan, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

UIST '21

Sponsor:

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

October 10 - 14, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

252
Total Citations
View Citations
1,236
Total Downloads

Downloads (Last 12 months)531
Downloads (Last 6 weeks)56

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Oo ZKAWAI NMATSUMIYA NHAYASHIDA M(2024)Japanese Teachers’ Attitudes and Self-Efficacy Toward Inclusive Education: Three-Way ANOVA and Regression AnalysesJournal of Education for Sustainability and Diversity10.57142/jesd.v3i1.6123:1(182-210)Online publication date: 29-Jul-2024
https://doi.org/10.57142/jesd.v3i1.612
Czeglédi ISpecziár APreiszner BBoros GBánó BMozsár ATakács PErős T(2024)Stable isotope analysis reveals diet niche partitioning between native species and the invasive black bullhead (Ameiurus melas Rafinesque, 1820)NeoBiota10.3897/neobiota.94.12249694(57-77)Online publication date: 26-Jul-2024
https://doi.org/10.3897/neobiota.94.122496
Miura TOkochi NMatsuo MYabu KKatagiri AYasu KSakajiri MIfukube T(2024)Examining How Interaural Differences Owing to Head Rotation during Walking Improve the Distance of Auditory Obstacle Perceptions among Individuals with Visual Impairment: A Case Study in Small-Scale Blind GroupLife10.3390/life1403035614:3(356)Online publication date: 7-Mar-2024
https://doi.org/10.3390/life14030356
Gyuris RSzabó ÁLászló AGutermuth ÁSörös C(2024)An Evaluation of Insecticidal Trunk Injections for the Control of the European Cherry Fruit Fly Rhagoletis cerasi L. (Diptera: Tephritidae)Horticulturae10.3390/horticulturae1003027810:3(278)Online publication date: 14-Mar-2024
https://doi.org/10.3390/horticulturae10030278
Tomes JFleischer PKubov MFleischer P(2024)Soil Respiration after Bark Beetle Infestation along a Vertical Transect in Mountain Spruce ForestForests10.3390/f1504061115:4(611)Online publication date: 28-Mar-2024
https://doi.org/10.3390/f15040611
Donato RContillo ACampana GRoccato MGonçalves ÓPavan A(2024)Visual Perceptual Learning of Form–Motion Integration: Exploring the Involved Mechanisms with Transfer Effects and the Equivalent Noise ApproachBrain Sciences10.3390/brainsci1410099714:10(997)Online publication date: 30-Sep-2024
https://doi.org/10.3390/brainsci14100997
Panfili MGuicciardi o Guizzardi SFrapiccini ETruzzi CGirolametti FMarini MSantojanni AAnnibaldi AIlluminati SColella S(2024)Influence of Contaminants Mercury and PAHs on Somatic Indexes of the European Hake (Merluccius merluccius, L. 1758)Animals10.3390/ani1420293814:20(2938)Online publication date: 11-Oct-2024
https://doi.org/10.3390/ani14202938
Arriola AAl Saify IWarner NHerzke DHarju MAmundsen PEvenset AMöckel CKrogseth I(2024)Dechloranes and chlorinated paraffins in sediments and biota of two subarctic lakesFrontiers in Toxicology10.3389/ftox.2024.12982316Online publication date: 16-May-2024
https://doi.org/10.3389/ftox.2024.1298231
Ota TMatsumoto KAoyama KAmemiya TNarumi TKuzuoka H(2024)Ankle tendon electrical stimulation to enhance sensation of walking on a slope in walking-in-placeFrontiers in Virtual Reality10.3389/frvir.2024.14562025Online publication date: 23-Oct-2024
https://doi.org/10.3389/frvir.2024.1456202
Liu FQiu KWang HDong YYu D(2024)Decreased wrist rotation imitation abilities in children with autism spectrum disorderFrontiers in Psychiatry10.3389/fpsyt.2024.134987915Online publication date: 18-Apr-2024
https://doi.org/10.3389/fpsyt.2024.1349879
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents