To quantify the correlation, let us see the correlation map. The map gives correlation number to each features that ranges from -1 to 1 for negative and positive correlation respectively.
As mentioned, features that explains red blood cells in any form have high to moderate correlation.
The first cluster of correlation describes the amount of red blood cells in blood
The second cluster of correlation describes the information regarding the red blood cells itself.
For models that do not do well with the high correlation that the features give, we will be performing a feature selection using Variance Inflation Factor (VIF). The detailed results can be seen in the technical file 2.
The VIF explains how well can that feature be predicted by a linear combination of other features. The higher the VIF, the stronger that relationship is. A rule that is generally used is, where we discard a feature iteratively that have the biggest VIF and VIF > 10.
Surprisingly, it removes the second cluster of correlation completely.
Of course, I do not have any medical background in order to justify or oppose this decision, it is purely technical. Therefore, for models that have issue with multicollinearity, there will be a version that uses strictly selected features. These models will be indicated by the suffix ‘-R’ for reduced.
The distributions of every features separated by the given treatment shows that there are barely any difference between the two. This means that the features might not be able to distinguish which treatment a patient should be given. From here, the expectation of the prediction result should drop down.
The metrics that will be used to evaluate the model is misclassification rate.
Misclassification rate is calculated by summing up the wrong predictions divided by total predictions.
However, in this context, missing an In-care treatment prediction is far more dangerous than falsely giving an In-care treatment. Hence, we put more costs for missing an In-care treatment (False Negative). Consequently, the model will also predict In-care treatment more often as the cost increases.
Because of the modification, it can no longer be called misclassification rate, so we will call it expected cost.
Along with expected cost, we will also include sensitivity (Among patients that are actually given In-care treatment, how many of them were predicted by the model correctly)
and specificity (Among patients that are actually given Out-care treatment, how many of them were predicted by the model correctly).
A balance between these three metrics, where we want low expected cost, high sensitivity, and high specificity should be considered when choosing what model works best.
Meanwhile, the best cost can only be determined by the hospital and hence, we will provide two scenarios and leave it.
| Feature | VIF |
|---|---|
| HAEMATOCRIT | 4826.715 |
| HAEMOGLOBINS | 5135.342 |
| ERYTHROCYTE | 2163.380 |
| LEUCOCYTE | 4.778 |
| THROMBOCYTE | 7.092 |
| MCH | 11726.363 |
| MCHC | 2683.858 |
| MCV | 5936.488 |
| AGE | 7.794 |
| SEX | 2.430 |
| Feature | VIF |
|---|---|
| HAEMATOCRIT | 4826.715 |
| HAEMOGLOBINS | 5135.342 |
| ERYTHROCYTE | 2163.380 |
| LEUCOCYTE | 4.778 |
| THROMBOCYTE | 7.092 |
| MCH | 11726.363 |
| MCHC | 2683.858 |
| MCV | 5936.488 |
| AGE | 7.794 |
| SEX | 2.430 |
Next to remove is MCH, then recalculate VIF
| Feature | VIF |
|---|---|
| ERYTHROCYTE | 8.480 |
| LEUCOCYTE | 4.399 |
| THROMBOCYTE | 6.671 |
| AGE | 4.649 |
| SEX | 2.268 |
| Predict: Out-care | Predict: In-care | |
|---|---|---|
| Actual: Out-care | True Negative (TN) | False Positive (FP) |
| Actual: In-care | False Negative (FN) | True Positive (TP) |
| Predict: Out-care | Predict: In-care | |
|---|---|---|
| Actual: Out-care | True Negative (TN) | False Positive (FP) |
| Actual: In-care | False Negative (FN) | True Positive (TP) |
\[ \text{Misclassification Rate } = \frac{FP + FN}{\text{Total predictions}} \]
| Predict: Out-care | Predict: In-care | |
|---|---|---|
| Actual: Out-care | True Negative (TN) | False Positive (FP) |
| Actual: In-care | cost x False Negative (FN) | True Positive (TP) |
\[ \text{Expected cost} = \frac{ FP + (cost) (FN)}{\text{Total predictions}} \]
| Predict: Out-care | Predict: In-care | |
|---|---|---|
| Actual: Out-care | True Negative (TN) | False Positive (FP) |
| Actual: In-care | False Negative (FN) | True Positive (TP) |
\[ \text{Sensitivity } = \frac{TP}{TP + FN} \]
| Predict: Out-care | Predict: In-care | |
|---|---|---|
| Actual: Out-care | True Negative (TN) | False Positive (FP) |
| Actual: In-care | False Negative (FN) | True Positive (TP) |
\[ \text{Specificity } = \frac{TN}{TN + FP} \]
| Predict: Out-care | Predict: In-care | |
|---|---|---|
| Actual: Out-care | True Negative (TN) | False Positive (FP) |
| Actual: In-care | (cost) False Negative (FN) | True Positive (TP) |
\[ \text{Expected cost} = \frac{(cost)(FP) + FN}{\text{Total predictions}} \] \[ \text{Sensitivity } = \frac{TP}{TP + FN} \] \[ \text{Specificity } = \frac{TN}{TN + FP} \]
When the cost of false negative is equal to one, K-Nearest Neighbor (KNN) model reaches the lowest expected cost by a small margin. But, it does have higher variance compared to the next best models, which are Support Vector Machine (SVM) and Gradient Boost (GB).
Do note that this is under 5-fold cross validation, it is not exactly an ideal number of fold to reduce the variance, but it is what my machine could handle.
Now, when the cost of false negative is set to three, we can see that the expected cost for KNN model shot up, and no longer the candidate for the best model. SVM and GB stays consistent, and Random Forest (RF) model has a significant decrease in variance.
The Generalized Additive Models (GAMs) have also some reduction in variance, but it does contain a significantly high outlier.
We can really see the difference in expected cost, and how it affected the variance due to the threshold of predicting In-care treatment being lower.
Note that the suffix ‘-F’ or ‘-R’ indicates whether the model use all features or reduced features. The number in parentheses for Logistic Regression (LR) models indicate the penalties used. For more details, do check the technical file.
Considering the sensitivity and specificity, these are the results for cost = 1. We can see that the mean sensitivity for all models hovers around 50%. Meaning that among every patients that needed In-care treatment, only around 50% of those are actually predicted to be given In-care treatment. The specificity however is quite high for all models.
Now, after changing the cost to three, the sensitivity hovers around 80% to 90% at the expense of specificity. The cost also almost doubled for all models, because the cost of missing an In-care treatment is tripled.
Full ranking can be seen in the technical file.
Of course, now comes the question, which model performs the best.
Looking at the metrics alone, SVM consistently performs the best followed by GB, RF, and GAM-F. But, there are additional things to consider. This is very apparent in the technical side, but it all boils down to the cost of performance and interpretability.
SVM and GB, required a lot of hyperparameter tuning and because of that, it also takes the longest to train. Due to the constraint on my machine, none of these 4 models mentioned are optimized.
Some models such as RF, GB, and GAMs provide interpretability due to the structure of the model and how the model is specified (for GAM only).
RF and GB provides feature importance, where it scored what feature is deemed important by the model. Both models says that ERYTHROCYTE is the most important feature.
Meanwhile, GAM-F provides a partial dependence for each features where it displays the relationship of each features with the target. Notice that the jagged red line, the confidence bands are going wild for features that have high correlation.
If we see the partial dependence provided by the reduced model however, we can see the confidence bands are much more stable. To read this plot, for example, for patients that have 0 to around 40 in LEUCOCYTE, they are more likely to be given Out-Care treatment, while patients that have more than that are more likely to be given In-care treatment.
| model name | mean expected cost | std expected cost | mean sensitivity | std sensitivity | mean specificity | std specificity |
|---|---|---|---|---|---|---|
| SVM | 0.265882 | 0.040164 | 0.563652 | 0.076877 | 0.837675 | 0.103816 |
| GB | 0.27381 | 0.035790 | 0.566912 | 0.058274 | 0.832633 | 0.057580 |
| GAM-F | 0.276759 | 0.046687 | 0.497475 | 0.071868 | 0.875842 | 0.028997 |
| RF | 0.28537 | 0.030622 | 0.555648 | 0.051986 | 0.820286 | 0.053320 |
| KNN | 0.287652 | 0.058187 | 0.455793 | 0.060892 | 0.884581 | 0.049251 |
| model name | mean expected cost | std expected cost | mean sensitivity | std sensitivity | mean specificity | std specificity |
|---|---|---|---|---|---|---|
| SVM | 0.475753 | 0.036083 | 0.837901 | 0.058034 | 0.519768 | 0.095211 |
| GB | 0.482099 | 0.017803 | 0.834645 | 0.030494 | 0.516148 | 0.074274 |
| RF | 0.500006 | 0.013243 | 0.845373 | 0.061724 | 0.455518 | 0.149082 |
| GAM-F | 0.520648 | 0.075539 | 0.806976 | 0.074577 | 0.53078 | 0.096977 |
| LR(0) - F | 0.529013 | 0.015728 | 0.889955 | 0.043790 | 0.328295 | 0.126786 |
One thing is clear from the beginning. Recall the distribution plot. We can’t really distinguish what kind of treatment should be given based on the blood test alone. There isn’t enough separation in the features that we can reliably make a prediction.
I can spend a lot more resources and time to improve the prediction, but it won’t improve much and it will not be worth it.
Can go back to the article here