Development, evaluation, and validation of machine learning models for COVID-19 detection based on routine blood tests

Introduction

To date, at eight months post-outbreak, the coronavirus disease (COVID-19) caused by the SARS-CoV-2 coronavirus has infected more than 20 million people and has resulted in approximately one million deaths worldwide. To manage this unprecedented pandemic emergency, the early identification of patients and of infectious people is extremely important due to the fact that this disease, unlike others caused by coronaviruses (e.g., SARS, MERS), can coexist in a host organism without causing any symptoms, or it can produce very mild and non-characteristic symptoms in — nevertheless — infectious subjects [1]. To identify SARS-CoV-2 infections, the instrument of choice, or the gold standard, is the molecular test performed using the reverse polymerase chain reaction (PCR) or the reverse transcriptase–PCR (RT-PCR) technique. However, the execution of the test is time-consuming (at no less than 4–5 h under optimal conditions), requires the use of special equipment and reagents, the involvement of specialized and trained personnel for the collection of the samples, and relies on the proper genetic conservation of the RNA sequences that are selected for annealing the primers [2]. In addition, for these pre-analytical vulnerabilities [3], the RT-PCR test’s accuracy, and especially its sensitivity (i.e., its ability to avoid false negatives), is far from ideal. A recently published article in the New England Journal of Medicine suggests that a reasonable estimate for the sensitivity of this test is 70% [4].

To improve our diagnostic capabilities, in order to contain the spread of the pandemic, the data science community has proposed several machine learning (ML) models, recently reviewed in [5]. Most of these models are based on computed tomography (CT) scans or chest X-rays [5], [6], [7], [8], [9]. Despite the reported promising results, some concerns have been raised regarding these and other works, especially in regard to solutions based on chest X-rays, which have been associated with high rates of false-negative results [10]. On the other hand, solutions based on CT imaging, although accurate, are affected by the characteristics of this modality: CTs are costly, time-consuming, and require specialized equipment; thus, approaches based on this imaging technique cannot reasonably be applied for screening exams. Although various clinical studies [11], [12], [13] have highlighted how blood test-based diagnostics might provide an effective and low-cost alternative for the early detection of COVID-19 cases, relatively few ML models have been applied to hematological parameters [14], [15], [16], [17], [18].

To overcome the above limitations, and following a successful feasibility study performed on a smaller dataset [19], we developed different classification models by applying ML techniques to blood-test results that are generally available in clinical practice within minutes (under emergency conditions, at even less than 15 min) and are only a fraction of the cost of the RT-PCR test and CT imaging (i.e., a few euros). As we will show, routine blood tests can be exploited by our method to diagnose COVID-19 patients in low-resource settings, in particular, where there is a shortage of RT-PCR reagents, such as during a pandemic peak. On the other hand, the developed method can also be used as a complement to the RT-PCR test in order to increase the sensitivity of the latter or to provide its interpreters a sort of pre-test probability to compute NPV and PPV. Furthermore, the rapid blood-test results could be a valuable — although non-conclusive — indication for the early identification of COVID-19 patients, resulting in their better management/isolation while waiting for the gold standard results.

Materials and methods

In this section, we describe the datasets and statistical methods used to train and validate the ML models. The reporting follows the TRIPOD Guideline for Model Development and Validation [20]. The study protocol (BIGDATA-COVID19) was approved by the Institutional Ethical Review Board in agreement with the World Medical Association Declaration of Helsinki.

Data description

OSR dataset

The main dataset used for this study (the OSR dataset) consisted of routine blood-test results performed on 1,925 patients on admission to the ED at the San Raffaele Hospital (OSR) from February 19, 2020, to May 31, 2020. In order to control for potentially confounding pathologies and other sources of bias, such as insufficient data availability, in ML development, 301 (15.6%) patients, admitted between February and April, were excluded from further analysis. All patients admitted during May 2020, on the other hand, were considered for the study, to have a balanced number of patients also from the late portions of the time frame considered.

For each case, COVID-19 positivity was determined based on the result of the molecular test for SARS-CoV-2 performed by RT-PCR on nasopharyngeal swabs. On a set of 165 uncertain cases, we also used the result of chest radiography and X-rays to improve over the sensitivity of the RT-PCR test [21], [22], [23], [24]. Uncertain cases were identified through two different methods: either patients who resulted positive within 72 h after a first negative test and were admitted as inpatients despite this test result; or patients who, despite having a negative test, had an hematochemical profile more similar to positive patients, as determined through multi-variate clustering based on a set of COVID-19 characteristic biomarkers [12] (aspartate aminotransferase [AST], lymphocytes, calcium, lactate dehydrogenase [LDH], PCR, white blood cells [WBC], D-dimer [XDP], fibrinogen). Of the 165 uncertain cases, only 52 of them have been considered as positive after comparison with the radiologic gold standard, while the remaining 113 were considered as negative (having a double negative test from both the RT-PCR and the radiologic gold standard): this results in an estimate of 93% sensitivity of the RT-PCR with respect to the composite ground truth.

Therefore, the OSR dataset consisted of a total of 1,624 cases: 786 of them received a positive diagnosis (48%) and 838 were negative cases (52%).

As covariate features, for each case, the patient’s age and gender, the presence of COVID-19 related symptomatology at admission (dyspnea, pneumonia, pyrexia, sore throat, influenza, cough, pharyngitis, bronchitis, generalized illness), and a set of 69 hematochemical values from laboratory tests were considered. The list of the analytes and instruments are reported in Table 1. The laboratory blood tests were performed according to the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) recommendations [25].

Table 1:

Complete list of the analyzed features in the OSR dataset.

Category	Instrument/sample	Parameter	Acronym	Unit of measure	COVID-specific features	CBC features	Missing rate, %
Hematological	Sysmex XE 2100/whole blood	White blood cells	WBC	109/L	X	X	2.4
		Red blood cells	RBC	1012/L	X	X	3.6
		Hemoglobin	HGB	g/dL	X	X	2.4
		Hematocrit	HCT	%	X	X	2.4
		Mean corpuscular volume	MCV	fL	X	X	3.6
		Mean corpuscolar hemoglobin	MCH	pg/Cell	X	X	3.6
		Mean corpuscolar hemoglobin concentration	MCHC	g Hb/dL	X	X	2.4
		Erythrocyte distribution width	RDW	CV%	X	X	3.7
		Platelets	PLT	109/L	X	X	3.6
		Mean platelet volume	MPV	fL	X	X	5.9
		Neutrophils count, (%)	NE	%	X	X	18.9
		Lymphocytes count, (%)	LY	%	X	X	15.2
		Monocytes count, (%)	MO	%	X	X	15.2
		Eosinophils count, (%)	EO	%	X	X	15.2
		Basophils count, (%)	BA	%	X	X	15.2
		Neutrophils count	NET	109/L	X	X	15.2
		Lymphocytes count	LYT	109/L	X	X	15.2
		Monocytes count	MOT	109/L	X	X	18.9
		Eosinophils count	EOT	109/L	X	X	15.2
		Basophils count	BAT	109/L	X	X	18.9
Coagulation	STA-R MAX/Plasma sample	Prothrombin time (INR)	PTINR	INR			31.0
		Activated partial thromboplastin time (R)	PPTR	Ratio			31.5
		Fibrinogen	FG	mg/dL			70.2
		D-dimer	XDP	µg/mL			70.4
Biochemical	Cobas 6000 Roche/serum sample	Glucose	GLU	mg/dL	X		3.4
		Creatinine	CREA	mg/dL	X		2.4
		Urea	UREA	mg/dL	X		37.0
		Direct bilirubin	BILD	mg/dL			23.3
		Indirect bilirubin	BILIN	mg/dL			23.3
		Total bilirubin	BILT	mg/dL			25.3
		Alanine aminotransferase	ALT	U/L	X		3.1
		Aspartate aminotransferase	AST	U/L	X		3.2
		Alkaline phosphatase	ALP	U/L	X		23.7
		Gamma glutamyltransferase	GGT	U/L	X		24.5
		Lactate dehydrogenase	LDH	U/L	X		13.2
		Creatine kinase	CK	U/L	X		60.3
		Sodium	NA	mmol/L	X		3.9
		Potassium	K	mmol/L	X		2.7
		Calcium	CA	mmol/L	X		3.8
		C-reactive protein	CRP	mg/L	X		5.5
		NT-proB-type natriuretic peptide	PROBNP	pg/mL			91.1
		Troponin T	TROPOT	ng/L			62.8
		Interleukin 6	IL6	pg/mL			92.2
Rapidpoint 500 (Siemens Healthcare)	Hemogasanalysis, venous blood gas	pH	PHPOC	U			18.5
		Carbonic anhydride (pCO2)	CO2POC	mmHg			22.4
		Oxygen (pO2)	PO2POC	mmHg			22.4
		Bicarbonates	BICPOC	mmol/L			18.7
		Standard calculated bicarbonates	BISPOC	mmol/L			23.0
		Base excess	BEPOC	mmol/L			22.8
		Actual base excess	BEEPOC	mmol/L			18.9
	CO-oxymetry	Hematocrit (POC)	HCTPOC	%			22.7
		Total oxyhemoglobin	THBPOC	g/dL			22.8
		O2 saturation	SO2POC	%			18.3
		Oxyhemoglobin/Total hemoglobin	FO2POC	%			18.6
		Carboxyhemoglobin	FCOPOC	%			18.8
		Methemoglobin	METPOC	%			22.5
		Deoxyhemoglobin	HHBPOC	%			18.8
	Oxygenation	Bound O2 maximum concentration	BO2POC	mL/dL			23.8
		Total oxygen	CTOPOC	mL/dL			20.9
		Inspired oxygen fraction	FIOPOC	mL/dL			67.4
		Inspired O2/O2 ratio	OFIPOC	Ratio			64.0
	Electrolytes POC	Sodium (POC)	NAPOC	mmol/L			22.5
		Potassium (POC)	KPOC	mmol/L			22.4
		Chloride (POC)	CLPOC	mmol/L			22.7
		Ionized calcium (POC)	CAPOC	mmol/L			23.1
		Standard ionized calcium (POC)	CASPOC	mmol/L			23.2
		Anion gap	ANGPOC	mmol/L			19.6
		Glucose blood gas	GLUEMO	mg/dL			18.6
		Lactate (POC)	LATPOC	mmol/L			18.5
Additional information		Age	Age	Years	X	X	0
		Gender	Sex	Male/Female	X	X	0
		COVID-19 suspect (patient suffers from COVID-19 specific symptoms at triage)	Suspect	Yes/No	X	X	0
Target		COVID-19 positivity	Target	Positive/Negative	X	X	0

The demographic and clinical characteristics of the two different groups of COVID-19 patients are summarized in Figures 1 and 2.

Figure 1:

Demographic feature (gender and age) distributions for positive and negative cases.

The blue and orange areas correspond to negative and positive cases, respectively.

Figure 2:

Violin plots depicting the distributions of eight relevant features in the OSR dataset (selected for their predictivity toward COVID-19). The blue and orange areas correspond to negative and positive cases, respectively.

The missing data rate for each of the examined features is reported in Table 1. In order to reduce the bias due to imputation, we discarded all the features with a missing data rate greater than 75%. Thus, among the 1,624 cases in the OSR dataset, 1,189 (73%) cases had at least 75% of the attributes; while 1,324 (82%) cases had complete data for the CBC features.

From the complete OSR dataset, we obtained two other datasets by selecting two relevant subsets of the features (thus, the three datasets share the same set of patients):

1. A dataset consisting of the 34 features under the column header “COVID-specific features” (see Table 1), denoted as the COVID-specific dataset.

2. A dataset consisting of the 21 features under the column header “CBC features” (see Table 1), denoted as the CBC dataset.

External datasets

In addition to the previously described datasets, all obtained from the OSR dataset, we considered two external datasets for the internal–external validation and for the external validation of the models.

The first dataset, the Istituto Ortopedico Galeazzi (IOG) dataset, was obtained from blood samples collected at the ED of the IOG of Milan between March 5, 2020, and May 26, 2020, and encompassed the parameters under the “COVID-specific features” column header (see Table 1). Notably, this hospital specializes in the diagnosis and treatment of musculoskeletal disorders and was not considered a destination of choice during the acute phase of the pandemic in the Milan area. Therefore, the patients were presumably of a different severity and were admitted for other reasons than pulmonary conditions with respect to OSR. The IOG dataset consisted of a total of 58 cases, 29 with negative swab results and 29 with positive swab results, and with the same features as the COVID-specific dataset. For the IOG and OSR, different instruments were used for the CBC and biochemical parameters; in particular, the iSysmex XN-2000 system was used instead of the Sysmex XE 2100 system for CBC counts and the Atellica® CH Analyzer (Siemens Healthineers) was used instead of the Roche COBAS 6000 system for the biochemical parameters.

The second dataset (the 2018 dataset) was obtained from blood samples collected at the OSR in November 2018 from 54 randomly chosen patients. These were obviously negative for COVID-19: 20 (37%) of them were specifically chosen to act as confounding cases, as they exhibited pneumonia-like symptoms.

Results

The results of the ML models on the three datasets (OSR, COVID-specific, CBC) are reported in Table 2.

The receiver operating characteristic (ROC) curves of the best model (in terms of the highest AUC) for each of the three datasets is reported in Figure 3. The ROC curves for all models (on each of the three datasets) are reported in Supplemental Figures 1, 2 and 3. The feature importance scores, which were computed in order to enable the interpretability of the developed models, are reported in Figure 4 and in Supplementary Figure 13 for the best model of each of the three datasets. The positive predictive value (PPV)-sensitivity curves are reported in Supplementary Figures 4, 5 and 6, while the calibration curves are reported in Supplementary Figures 7, 8, and 9, and the PPV/NPV prevalence curves are reported in Supplementary Figures 10, 11 and 12.

Figure 3:

Receiver operating characteristic curves for the best models (in terms of AUC), for each of the three considered datasets (OSR, COVID-specific, CBC).

For the CBC dataset we report the ROC curve for both RF and KNN as they had equal AUC (see Table 2).

Figure 4:

Feature importance scores for the random forest algorithm trained using the OSR dataset (on the left) and the k-nearest neighbors’ algorithm trained using the COVID-specific dataset (on the right).

The results for the internal–external validation and the external validation (specificity only) procedures are reported in Table 3. In this table we highlight the results of the models that obtained the best performance in the internal validation (KNN for the COVID-specific dataset; and both KNN and RF for the CBC dataset). Specifically, in the first four columns we report the results of the internal-external validation (in terms of accuracy, sensitivity, specificity and AUC), while in the last column we report the results of the external validation (in terms of specificity). In Table 3, we also report on the performance of the models for asymptomatic patients and symptomatic patients, as described in the Methods section.

Discussion

The unprecedented worldwide public health emergency caused by the COVID-19 pandemic has motivated different research groups to develop ML applications with the aim of automating — at least partially — the diagnosis or screening of COVID-19.

Nonetheless, only a few studies have focused on the development of ML models based on routine blood exams. Formica et al. [13] developed a CBC-based ML model, reporting 83% sensitivity and 82% specificity; however, the analysis was based on a small sample (171 patients) collected in a limited time frame (between March 7 and March 19, 2020). Banerjee et al. [32] applied ML methods to a public dataset of CBC data encompassing 598 cases of which only 39 cases were COVID-19 positive; the authors report good specificity (91%) but very low sensitivity (43%), thus making the proposed model unsuitable for early detection tasks. Further, this work presents some major limitations affecting replicability and generalizability, as the authors do not provide any information regarding how the values of the considered features were measured (analytical instruments, analytical principle, and units of measurement). Avila et al. [33] used the same dataset considered in [32] to develop a Bayesian model, reporting 76.7% sensitivity and specificity. Notably, the authors report a number of complete instances (510) which is different from that reported in [32]. Joshi et al. [34] developed a logistic regression model trained using CBC data on a dataset of 380 cases, reporting good sensitivity (93%) but low specificity (43%).

More in general, a recent critical survey [5] raised some concerns about these and other evaluated studies (most of which have not yet undergone peer-review), noting the possibility of high rates of bias and over-fitting, and little compliance with reporting and replication guidelines [18].

Finally, a recent study Yang et al. [35], considered the development of a Gradient Boosting model on a set of 3,356 patients (42% COVID-19 positive) using a set of 27 parameters encompassing both blood count and biochemical parameters, achieving 0.85 AUC, and also reporting a comparable result (AUC 0.84) for validation on an external dataset. This work can be viewed as similar but complementary with respect to the results that we report, both in terms of considered features and used laboratory instrumentation (the authors used the UniCel DXH 800 analyzer for the CBC features, and Siemens ADVIA XPT analyzers for biochemical parameters). Indeed, compared with the parameters considered in this study, the authors of [35] considered albumin, total protein, magnesium, ferritin and globulin; but lacked a set of parameters (some of which known to be significantly altered in COVID-19 patients), such as creatinine (CREA), aspartate aminotransferase (AST), alanine aminotransferase (ALT), γ-glutamyl transferase (GGT), creatine kinase (CK), potassium (K), interleukin-6 (IL6), NT-proB-type natriuretic peptide (ProBNP), total (BilT) and direct (BilD) bilirubin, all coagulation tests, hemogasanalysis and CO-oxymetry parameters. We think that this complementarity in the two studies could lend support to the usefulness of blood tests as an alternative approach for COVID-19 diagnosis.

To overcome the limitations of the above models, we applied the ML methodology to routine blood examination outcomes, which are usually available for inpatients and for patients admitted to the ED in shorter time frames and at much lower cost than both molecular tests and radiological exams. In this endeavor, we addressed three subtasks: (1) detecting COVID-19 from a full battery of hematochemical tests, commonly collected from suspected respiratory tract disease patients (OSR dataset); (2) detecting COVID-19 from only a restricted subset of parameters known to be altered in COVID-19 patients (COVID-specific dataset); and (3) detecting COVID-19 from a very small subset of hematological parameters (CBC and the WBC differential) representing the basic routine blood examinations, usually also available in low-resource settings (CBC dataset). For each of the datasets described above, we applied five different models that were selected from among those that are more frequently adopted in medical ML.

These models achieve COVID-19 detection in different ways and exhibit good performance, although they are associated with different sensitivities and specificities. This makes them good candidates for embedding in an online service (we made available two tools, one for the general users - https://covid19-bloodtests-ml.herokuapp,con – and one for more technical ones – https://covid-19-blood-ml.herokuapp.com) in which doctors can specify their preferences (with respect to greater sensitivity, greater specificity, or a balanced performance [36] and needs according to their diagnostic purpose (i.e., screening, triage, or a secondary diagnosis), and thus gain an indication from the optimal model. In addition, the users can decide whether they want an indication from the system, irrespective of the confidence in the advice given, or if they would prefer only to be advised about high-confidence indications, as the three-way approach allows for. This approach was specifically developed to mitigate the risk of automation bias and the odds of machine-induced errors [31].

With respect to the patterns used to discriminate between positive and negative cases, the ML models identified, as the most predictive features, those parameters that are known to be significantly altered in COVID-19 patients [5], [37], [38] (see Figure 4 and Supplementary Figure 13). For instance, when applied to the OSR and COVID-specific datasets, the models identify lactate dehydrogenase (LDH), AST, C-reactive protein (CRP), and calcium as the most important features, while all the models also reported WBC and its corresponding differential as important. Also the patients’ age was reported by the models to be a significant predictor, which is consistent with the literature [5], where it was also found to be a significant predictor not only for prognostic, but also for diagnostic tasks. Notably, fibrinogen and cross-linked fibrin degradation products (XDPs), known to be associated with COVID-19 severity [39], were also considered by the model as being among the most important features when applied to the OSR dataset.

With respect to the calibration of the developed models, the good internal calibration of the models can be confirmed by the calibration curves in Supplementary Figures 7–9.

As can be seen in Table 3, the internal–external validation and the external validation procedures also achieved good results. In this respect, it is important to note that the validation procedures involved blood tests performed on different types of analytical instrumentation for the clinical chemistry tests (Siemens instead of Roche), although the CBC standardization was less problematic than the other tests. For this reason, as ML models exhibit poor performance when considering out-of-distribution samples [40], the goal of the internal–external validation process was to assess the capability of the models to generalize across different settings. All of the models showed good performance and, more specifically, good specificity. The models achieved good performances in symptomatic patients (with both the sensitivity and specificity at approximately 80%) and they performed even better in terms of specificity in asymptomatic patients (100% specificity), although the sensitivity was as low as 50% (see Table 3). Nevertheless, considering that the developed ML-based tests were based on low-cost and rapid blood-test examinations, the reported values can be considered good enough, specifically in regard to screening [16].

The external validation procedure also achieved very good results (at around 95% for all models in the standard version), but it should be noted that this only relates to COVID-19-negative patients, and hence, to specificity. Notably, in the external validation process, all five patients for which the models failed had symptoms that were compatible with COVID-19 disease.

As hinted at above, the outputs from our models can be used in different scenarios. They could be used together and combined with the molecular test to obtain a compound test with higher accuracy, and, most importantly, higher sensitivity regarding suspected cases, thus allowing for the identification of a larger number of COVID-19-positive patients so that they can be isolated and treated in a timely manner. Indeed, we can see in Table 3 that the sensitivity in symptomatic patients is adequate for this type of use. In the same vein, the models’ outputs could be used while waiting for the results from other tests, allowing for the timely and prudent management of suspected COVID patients, or in screening and pool-testing scenarios [41] where low accuracy is not a critical problem if a test such as a CBC can be performed frequently [42]. In Table 3 we can see that the model that was defined based on the smallest dataset (the CBC dataset) reaches 100% specificity in asymptomatic patients. Consequently, we are planning to use our model for epidemiological purposes on the blood donor population to estimate the prevalence of the condition in the asymptomatic population. On the other hand, the scenarios in which the results from our models replace those of the molecular tests address an emergency need, especially if the time to obtain the molecular test results is too long (due to a high demand for such tests in an outbreak area), if there is a shortage of materials (swabs or reagents) for any supply problem, or in poor health contexts or in contexts where there are serious structural deficiencies (such as in some developing countries or in a geographical area that is, in the meantime, affected by other socio-sanitary and humanitarian emergencies). In these situations where resources are limited and population-wide testing cannot be performed, CBC-based scores may help to pre-evaluate patients and activate COVID-19-specific pathways and molecular testing for patients with high scores independent of symptom severity. In the presence of suspected COVID-19 cases and high scores, logistical management can promptly activate isolation procedures [43].

Conclusions

All things considered, the ML models that we presented in this article achieved a performance that is comparable, although inferior, to RT-PCR [4], which is the current gold standard for COVID-19 diagnosis. Nevertheless, although our models are less accurate, they aim to be an additional tool available among those that, being much faster and cheaper than the current diagnostic reference tests, can be used for the screening of whole populations. This use can facilitate the shift in testing strategy that, grounding on a faster, although less accurate, identification of infected individuals, is said to have a positive potential in slowing the virus’ spread and contributing for the safe reopening of schools and workplaces [44].