Outcome Measures in Clinical Trials for Multiple Sclerosis

Limitations and Caveats

Despite general acceptance of the EDSS, there are many limitations and caveats (summarized in Table 3) [16]. First of all, EDSS holds high intra- and inter-rater variability [10, 11, 17–19]. This can be explained by the subjective nature of the neurological examination itself on which the EDSS is largely based, particularly in the lower EDSS range. Also, complex and ambiguous scoring rules for the FS probably explain some of the variability.

Non-linearity of the EDSS is another limitation (visualized in Fig. 1). The staying time in the middle scores is shortest and this results in a bimodal distribution with peaks at 1.0–3.0 and 6.0–7.0 [7, 20]. It means that the rate of progression as assessed by the EDSS varies depending on baseline score. Furthermore, responsiveness of the EDSS is limited [16, 21]. Scores higher than 4.0 are less influenced by changes in FS scores. For example, development of a paresis in a patient with an EDSS of 6.0 will not result in a higher EDSS. Conversely, EDSS would have changed with a baseline EDSS of 4.0.

The non-linearity and limited responsiveness should both be accounted for when interpreting changes over time [22]. Nevertheless, EDSS change is often presented without accounting for the baseline score. As a result, statistically significant change may erroneously be presented as clinically relevant and vice versa. An increasingly used clinically meaningful change is a change of 1.0 or more if EDSS at baseline was 0 to 5.5, and 0.5 or more for higher baseline EDSS scores. This is more driven by reproducibility data than by clinical relevance data.

Because the EDSS is an ordinal scale, non-parametric statistics should be used in statistical analysis. This implies that significant differences between groups can be calculated, but the magnitude of differences cannot. In line with this, results should not be presented with means and standard deviation, but with median values and interquartile ranges. Also, a caveat of numeric values is that they might give the false impression of being precise.

Another limitation is that clinical phenotypes are unevenly distributed across the EDSS. Because ambulatory dysfunction is one of the main characteristics in patients with progressive disease (SPMS and PPMS), these patients represent a larger proportion in the range of 4.0–7.5 [23, 24].

Lastly, several domains are not (sufficiently) assessed. Examples are cognitive function, mood, energy level and quality of life. Symptoms in these domains are frequently observed in MS patients and they may influence FS scores, ambulation and ADL function.

Suggested Improvements

During the International Conference on Disability Outcomes in MS (ICDOMS) that was held in 2011, several refinements for the EDSS were suggested to improve performance [25]. Firstly, a standardized script for questioning patients (which is necessary for some FS scores) might improve reliability and decrease the risk of unblinding in clinical trials (an example of the Neurostatus form may be found on http://www.neurostatus.net/). Secondly, simplification of scoring rules might reduce intra- and inter-rater variability. Thirdly, long-term disability worsening should be assessed with confirmation of EDSS worsening at 6 rather than 3 months. The main reason for this is that relapses may improve beyond 3 months, and thus EDSS worsening may be temporary [26]. Fourthly, streamlining of the EDSS might be achieved by finding the components of FS that contribute most to confirmed worsening of disability and omitting the other less informative components. Lastly, modification of the EDSS to improve linearity of measurement will facilitate statistical analysis and clinical understanding.

Whatever its limitations, the EDSS will probably continue to be the main disability measure for the near future because of the vast experience with it and the possibility of making historical comparisons. Until we have better alternatives, clinical assessment can be improved by using the EDSS in conjunction with other measures.

Limitations and Caveats

There are several caveats when using relapses as an outcome measure (summarized in Table 3). First of all, identification of a relapse is subjective. Ensuring perfect blinding for treatment is therefore essential. To limit subjectivity, a second assessment can be performed to objectify the relapse. The problem with this approach is that symptoms or signs may already have recovered, and recall bias of the patient and observer bias from the examiner may influence the second assessment [35].

Another caveat is that identification of a relapse largely depends on a patient reporting new symptoms. When a patient only reports new symptoms on scheduled visits and not spontaneously, the established relapse rate will be lower than in reality. In fact, increasing the number of visits in a trial period may increase the relapse rate [36].

An interesting phenomenon is that relapse rate is often remarkably high prior to inclusion into trials. Various explanations may be given for this [37, 38]. First of all, relapses in the preceding period of a trial are usually determined retrospectively and patients may over-report the exact number to qualify for inclusion. Secondly, the inclusion criterion of relapse rate is often high, meaning that only patients with very active disease are included. As a consequence, it can be expected that the relapse rate of these patients will decrease towards a disease average during the trial (i.e. regression to the mean). Thirdly, patients participating in a trial may do better merely because of a placebo effect or better comprehensive care during the trial. Lastly, during the natural course of MS the relapse rate will eventually decrease, independent of treatment [39]. These factors may obscure the interpretation of absolute relapse rate reduction in treatment trials.

The Original Components

Ambulatory function is tested with the timed 25-foot walk test (T25W, explained in Table 4). The T25W is a reliable test for patients with more severe gait impairment, because it primarily assesses walking speed. Assessing walking speed seems clinically relevant, because it relates to the capacity to perform outdoor activities important in daily life [54]. For patients with mild gait impairment, the T25W may not be sensitive enough to detect abnormalities and because of that has a floor/ceiling effect [55]. For these patients, it may be more appropriate to assess walking endurance with longer walking distances; for example, with a 6-minute walking test [56].

Hand function is tested with the nine-hole peg test (9HPT, explained in Table 4). A change of 9HPT correlated with long-term disability [57].

The paced auditory serial addition task (PASAT, explained in Table 4) was originally included to cover the cognitive domain [58]. It measures processing speed and working memory, both of which are frequently affected functions in MS patients [59]. The test has moderate reliability and sensitivity for detection of cognitive impairment, and has limited responsiveness to change [60]. Furthermore, it requires a certain mathematical ability and has a clear ceiling effect [49, 61]. Finally, it is often disliked by patients because the time limit induces stress.

Candidate Components

A candidate cognitive test that may replace the criticized PASAT is the symbol digit modalities test (SDMT, explained in Table 4) [62, 63]. It measures information processing speed. The advantages of the SDMT are that it is easily administered, better tolerated by patients (probably because there is no time pressure) [64] and more robust and reliable than the PASAT [65, 66]. Moreover, the SDMT correlated more strongly with white matter abnormalities than PASAT [67, 68]. It also correlated with worsening of cognitive impairment [69, 70] and MRI abnormalities (atrophy measures in particular) [71, 72]. A limitation is that a patient has to have an intact visual system, which may be impaired in MS patients. Although there is a ceiling effect, it is less pronounced than for the PASAT. All points considered, the SDMT is probably a good replacement for the PASAT.

When the MSFC was developed, no data on suitable tests to assess visual function were available. In the past decade, various visual outcome measures for MS research have been studied [73]. Of these, the low-contrast letter acuity test (LCLA, explained in Table 4) may be a good candidate to add to the MSFC [74]. Results correlated with clinical phenotypes, MRI abnormalities and PROMs for visual impairment and quality of life (which supports clinical relevance) [75, 76]. Moreover, some clinical trials showed treatment effect on the LCLA in the active group compared with placebo [77].

Limitations and Caveats

There are several limitation and caveats of the MSFC (summarized in Table 3). A frequently postulated objection to the MSFC is that the overall score lacks a clear dimension, which hinders interpretability and therefore appears to be difficult for the interpreter to get familiar with the score. In other words, it is difficult to form a ‘mental picture’ of it [78]. This difficulty may be addressed by keeping the elements of the MSFC score separated instead of combining them into a single score. Nonetheless, comparison of subtest results between studies remains impossible due to the Z scores that obscure the meaning of crude scores.

Another problem is that results from the reference group strongly influence the Z scores of patients [79]. With that, assessing changes in time is problematic, because the overall score is influenced by variability between time points of both the reference and patient group. Consequently, it is impossible to determine if change is a result of statistical variance or true progression of disability [38].

A potential solution to some of the statistical caveats of Z scores might be to determine the minimal clinically relevant change [21, 80]. This means that change should be confirmed on a subsequent time point, preferably at 6 months (because of possible disability improvement after a relapse). This approach has been tested in a clinical trial dataset [45]. Sensitivity of worsening was found to be similar between MSFC and EDSS, and it correlated with other clinical and MRI outcome measures. However, the downside of this approach is that it will hamper sensitivity to change, which is of particular importance in patients with severe disability.

Despite its disadvantages, the MSFC is an appealing alternative for the EDSS. It can be performed within 20 minutes, covers three domains, has good intra- and inter-rater reliability and it results in a score on a continuous scale. The MSFC has been used as the primary outcome in a treatment trial in SPMS [49]. While MSFC progression was slowed, treatment effects were not observed with the EDSS. If the components are applied in a sensible way, the MSFC may be used as the primary endpoint in future clinical trials.

White Matter Pathology

MRI is sensitive to detect, characterize and quantify lesions in the white matter. It plays a fundamental role in the McDonald diagnostic criteria for MS to demonstrate dissemination in time and space in addition to clinical signs [2]. Radiological dissemination in space is defined as having at least one lesion in at least two typical (for MS) areas in the central nervous system. Dissemination in time is determined when at least one new lesion is demonstrated on a follow-up MRI, or if one asymptomatic gadolinium-enhancing and one non-enhancing lesion are demonstrated on the initial MRI.

The MAGNIMS workgroup recently proposed a revision of these criteria allowing even earlier diagnosis with MRI [116]. The value of MRI as a diagnostic tool is principally the high sensitivity to detect (past) disease activity. Formation of new T2HL and GdT1L may occur subclinically and are thus more frequently seen than clinical relapses [9, 117]. The moderate correlation of T2HL load with relapse rate [26, 118] and disability [119, 120] is possibly related to this phenomenon. Nevertheless, white matter pathology has predictive value for the clinical disease course. For example, patients with a CIS and a high T2HL load at baseline had an increased risk of reaching an EDSS of 3.0 [121]. Also, the presence of two or more GdT1L in patients treated with interferon-β predicted EDSS worsening at 15 years [122].

Because of the high sensitivity for detecting disease activity, MRI has been widely accepted as a secondary endpoint in clinical trials. Moreover, demonstrating efficacy on MRI lesions is crucial in the development of immunomodulatory treatments. Treatment effects on MRI could also act as a surrogate endpoint for clinical disease activity. A study supported this by showing that treatment effect on MRI activity explained >80% of the variance of treatment effect on relapse rate [123]. Other studies confirmed this by showing the related MRI effects on relapse rate and accumulation of disability worsening (up to 16 years) [124–126].

These classical MRI parameters largely depict (past) neuroinflammation in MS. However, the neurodegenerative aspect of MS is being increasingly studied with MRI. One reason for this is that with the current therapy we are now able to suppress neuroinflammation effectively, but the ultimate goal of therapy is prevention of neuronal tissue loss or, in the long run, to stimulate neuronal repair. Another reason is that neuropathological and MRI techniques have improved our insight into the underlying neurodegenerative processes of MS [127]. Consequently, measures that reflect these processes are more frequently used as secondary outcome measures. The most widely used neurodegenerative MRI measures are atrophy and PBH.

Atrophy

Brain volume loss in MS patients occurs considerable faster than in healthy people: 0.5–1.0% versus 0.1–0.3% brain volume loss per year [128, 129]. Atrophy may be found throughout the disease course, even in the early phases [130]. Remarkably, the atrophy rate of gray matter structures accelerates in patients with SPMS to 14-fold that of healthy persons [131]. Virtually all gray matter structures are affected, although variation exists between clinical phenotypes [132].

Brain volume can be visualized in various ways. The somewhat older measures assess loss of brain volume indirectly by measuring corpus callosum size [133], bicaudate ratio [72] and ventricular volumes [72, 133]. Also, whole brain volume can be measured directly with conventional MRI [72, 128]. Nowadays, segmentation of the brain into white and gray matter compartments or specific gray matter structures is possible and several automated methods reduced processing time [134–136].

The relationship between atrophy measures and clinical signs has been extensively investigated. Whole brain and gray matter atrophy correlated strongly with disability and cognitive impairment, both cross-sectionally and longitudinally [132]. These correlations existed throughout the disease course and clinical phenotypes. Atrophy of gray matter structures may even be more closely related to clinical signs than white matter lesions or whole brain atrophy [137]. Atrophy of several structures correlated remarkably strongly with certain clinical symptoms. For example, cerebellar gray matter atrophy correlated strongly with cerebellar symptoms and hand function [138], upper cervical cord area with ambulatory dysfunction [139], and hippocampal atrophy with memory deficits [140]. Thalamic volume showed a remarkably firm correlation with cognitive impairment [141]. Also, various atrophy measures showed predictive value for future disability and cognitive impairment [137, 142–144].

Furthermore, spinal cord volumes can be assessed, for which the upper cervical cord area is often used. Several studies showed a correlation between spinal cord volume loss and clinical disability [144–146]. It has also been correlated with long-term disability [147].

An extensive summary of clinical trials that used brain atrophy as a secondary endpoint may be found elsewhere [148, 149]. Noteworthy is a recent meta-analysis that showed that 75% of the variance of treatment effect on disability was explained by whole brain atrophy and T2HL [150]. Another meta-analysis found evidence that whole brain atrophy in patients that received immunomodulatory treatment was lower than in the placebo group [151].

Although volumetric measurements are appealing outcome measures, there are some caveats and limitations. Firstly, atrophy accumulates very slowly, which generally means that longer follow-up is needed to detect significant changes. Clearly, this accounts particularly for treatment effects on smaller structures, such as thalamic volume. Secondly, the short-term effect of immunosuppression on brain tissue may cause a decrease in brain volume due to resolution of inflammation. This volume loss is not a sign of neurodegeneration, because there is no loss of neuronal tissue. This is often referred to as ‘pseudo-atrophy’. Importantly, this effect may last up to 1 year after initiation of treatment [152, 153]. Thirdly, various physiological variations in the content of the intra- and extra-cellular compartments affect volumetric measurements [154]. Lastly, factors that are not MS-specific (such as dehydration, alcohol use, smoking, genetic variation, comorbidities and age) may influence brain volume [154].

Persisting Black Holes

Another MRI marker for neurodegeneration is formation of PBH. These lesions are often defined as non-enhancing T2HL with persisting signal intensity between that of the gray matter and the CSF on T1-weighted scans [155]. Approximately 30–40% of active T2HL will eventually evolve into PBH within 6–12 months [156]. The underlying neuropathology of PBH is severe and irreversible tissue damage [156]. Accumulation of PBH is associated with accrual of disability [157, 158]. Furthermore, the PBH load correlated with disability worsening over 10 years [159]. Some clinical trials found significant effects of treatment on the formation of PBH [160–163].

Several more advanced MRI techniques are potentially valuable outcome measures, although they need further research to clarify the exact relevance. Examples are functional MRI for analysis of functional connectivity [164], diffusion tensor imaging to examine brain tissue integrity [165] and magnetization transfer ratio MRI as a marker for brain myelin content [166, 167].