Introduction 1 2 3 4 5 6 7 8  SAPS 3 ADMISSION SCORE Box I: Box II: Box III:  SAPS 3 PROBABILITY OF DEATH  data available at ICU admission or shortly thereafter Methods and statistical analysis Primary variable selection 9 10 Missing values were coded as the reference or “normal” category for each variable. When dual data collection was used—maximum and minimum values recorded during a certain time period—missing maximum values of a variable were replaced by the minimum, if documented, and vice versa. Some regression imputations were performed if noticeable correlations to available values could be exploited. For a detailed description of data collection and handling, see Part 1 of this report. Selection of variables was done according to their association with hospital mortality, together with expert knowledge and definitions used in other severity of illness scoring systems. The objective of using this combination of techniques rather than regression-based criteria alone was to reach a compromise between over-sophistication of the model and knowledge from sources beyond the sample with its specific case mix and ICU characteristics. Cross validation 11  p 13 14 15 16 Reducing model complexity  p 12 Significant predictors (n=70) were in a second step entered into a logistic regression model. The criterion for a predictor to enter the model was homogeneity across the five model-building processes: in principle, predictors should enter the model in all five development sets, but depending on the frequency of the predictor in the samples, the magnitude of the effect, and medical reasoning, some predictors were included if they appeared in the model in at least three subsamples. An example is the presence of Acquired Immunodeficiency Syndrome (AIDS): it was selected as a comorbidity in only 81 patients (0.48%), but the mortality—without controlling for other variables—in these patients was 42%. By taking all the above steps to identify the set of predictors, although deliberately not using any formal numeric criterion, we reduced the complexity of the model to minimize the amount of overfitting: This process resulted in 61 item classes (representing 20 variables) remaining in the final model. Using the parameter estimates from the logistic regression as starting values, a multilevel model was applied in the next step, using patient characteristics as fixed effects and ICUs as a random effect. Estimates were again calculated for the five development sets (for both, patient and ICU -based development subsamples). At this stage it was checked if rounding of coefficients (which allows for an easier manual computation of the score) would influence results, which was found not to be the case. Consequently, this was the approach chosen for the final construction of the SAPS 3 admission score sheet. The stability of the processes of variable selection and reducing complexity was further checked by bootstraping with replacement the total sample 100 times, both at patient level and at ICU level. Predicting hospital mortality  g After finishing these steps of cross-validation, the final estimates for the selected predictors of the SAPS 3 score as well as the selected shrinkage procedure were then calculated from the total sample of patients.  g http://www.r-project.org Results 1 2 Five variables for evaluating Box I: age, co-morbidities, use of vasoactive drugs before ICU admission, intrahospital location before ICU admission, and length of stay in the hospital before ICU admission; Five variables for evaluating Box II: reason(s) for ICU admission, planned/unplanned ICU admission, surgical status at ICU admission, anatomic site of surgery, and presence of infection at ICU admission and place acquired; Ten variables for evaluating Box III: lowest estimated Glasgow coma scale, highest heart rate, lowest systolic blood pressure, highest bilirubine, highest body temperature, highest creatinine, highest leukocytes, lowest platelets, lowest hydrogen ion concentration (pH), and ventilatory support and oxygenation. Table 1  SAPS 3 admission scoresheet—Part 1 Box I          0  3  5 6 7 8 9 11 13 15 18 Age, years          <40    >=40<60    >60<70  >=70<75 >=75<80 >=80 Co-Morbidities            2)    3),4)  3)  5)    1)          <14     >=14<28 >=28       Intra-hospital location before ICU admission              Emergency room  Other ICU 6)      Use of major therapeutic options before ICU admission            Vasoactive drugs           Box II          0  3 4 5 6   9     ICU admission: Planned or Unplanned            Unplanned           Reason(s) for ICU admission please see Part 2 of the scoresheet                    Surgical status at ICU admission          Scheduled surgery    7) Emergency surgery        Anatomical site of surgery please see Part 2 of the scoresheet                    Acute infection at ICU admission             8) 9)         Box III 15 13 11 10 8 7 5 3 2 0 2  4 5  7 8      Estimated Glasgow Coma Scale (lowest), points 3–4   5  6   7–12 >=13             Total bilirubine (highest), mg/dL          <2   >=2<6 >=6         Total bilirubine (highest), µmol/L          <34.2   >=34.2<102.6 >=102.6         Body temperature (highest), Degrees Celsius      <35    >=35             Creatinine (highest), mg/dL          <1.2 >=1.2<2     >=2<3.5 >=3.5      Creatinine (highest), µmol/L 3–4   5  6    <106.1 >=106.1<176.8     >=176.8<309.4 >=309.4      Heart rate (highest), beats/minute          <120    >=120<160  >=160       Leukocytes (highest), G/L          <15 >=15            Hydrogen ion concentration (lowest), pH        <=7.25  >7.25             Plateletes (lowest), G/L  <20   >=20<50  >=50<100   >=100             Systolic blood pressure (lowest), mm Hg   <40  >=40<70   >=70<120  >=120             10), 11)   PaO2/FiO2<100 and MV   PaO2/FiO2>=100 and MV PaO2<60 and no MV   PaO2>=60 and no MV             The definition for all variables can be found in detail in Appendix C of the ESM. For names and abbreviations which are differing from those in the ESM, explanations are given below. Generally, it should be noted that no mutually exclusive conditions exist for the following fields: Comorbidities, Reasons for ICU admission, and Acute infection at ICU admission. Thus, if a patient has more than one condition listed for a specific variable, points are assigned for all applicable combinations. 1  2 3 4  5 6 7 8 9  10 2 2 11 Table 2 SAPS 3 admission scoresheet – Part 2 Box II – continued 12) 16 Reason(s) for ICU admission 13) –5 13) –4 3) 3    Neurologic: Coma, Stupor, Obtuned patient, Vigilance disturbances, Confusion, Agitation, Delirium 4 3) 5    Hepatic: Liver failure 6    Neurologic: Focal neurologic deficit 7    Digestive: Severe pancreatitis 9    Neurologic: Intracranial mass effect 10    All others 0 Anatomical site of surgery    Transplantation surgery: Liver, Kidney, Pancreas, Kidney and pancreas, Transplantation other –11    Trauma – Other, isolated: (includes Thorax, Abdomen, limb); Trauma – Multiple –8    Cardiac surgery: CABG without valvular repair –6    Neurosurgery: Cerebrovascular accident 5    All others 0 12) 13)  p 1 Logit = −32.6659 +ln(SAPS 3 score +20.5958) ×7.3068 and the probability of mortality by the equation:  logit logit Fig. 1  Distribution of the SAPS 3 admission score in the SAPS 3 database  2 p p 3 4 3 4 5 Fig. 2  Relationship between the SAPS 3 admission score and the respective probabilities of hospital mortality  Fig. 3  Columns:  squares:  circles:  Fig. 4  Columns:  squares:  circles:  Table 3  Performance of the model across major patient typologies Patient characteristics  GOF test Ĥ p GOF test Ĉ p O/E ratio 95% CI aROC Trauma patients 19.92 0.03 9.03 0.53 1.03 0.93–1.12 0.854 a 14.86 0.14 17.8 0.06 1.01 0.98–1.04 0.825 a 11.5 0.32 27.39 <0.01 0.97 0.90–1.03 0.825 a 4.97 0.89 12.88 0.23 1.00 0.95–1.05 0.809 b 8.57 0.57 14.77 0.14 1.00 0.97–1.02 0.846 c 8.4 0.59 11.76 0.3 1.00 0.96–1.05 0.786 d 15.21 0.12 7.11 0.72 1.02 0.97–1.07 0.77 GOF O/E CI aROC a b No infection c Community-acquired infection d Hospital-acquired infection Table 4  Performance of the model in the global sample and in different geographic areas  GOF test Ĥ p GOF test Ĉ p O/E ratio 95% CI aROC Australasia 15.25 0.12 8.09 0.62 0.92 0.85–0.99 0.839 Central, South America 78.01 <0.01 80.82 <0.01 1.30 1.23–1.37 0.855 Central, Western Europe 56.45 <0.01 47.89 <0.01 0.84 0.79–0.90 0.861 Eastern Europe 19.45 0.03 18.69 0.04 1.09 1.00–1.19 0.903 North Europe 2.44 0.99 2.34 0.99 0.96 0.83–1.09 0.814 Southern Europe, Mediterranean countries 14.18 0.16 20.78 0.02 1.02 0.98–1.05 0.834 North America 10.57 0.39 9.63 0.47 0.91 0.78–1.04 0.812 Global database 10.56 0.39 14.29 0.16 1 0.98–1.02 0.848 GOF O/E CI aROC Fig. 5  Observed-to-expected (O/E) mortality ratios by region. Observed-to-expected (O/E) mortality ratios are shown by region. Bars indicate 95% confidence intervals  5 5 Table 5  Customized SAPS 3 admission equations for the different geographic areas Area Equation GOF Ĥ p GOF Ĉ p O/E CI Australasia Logit=−22.5717 + ln (SAPS 3 score + 1) ×5.3163 10.43 0.40 2.20 0.99 1.00 0.93–1.07 Central, South America Logit=−64.5990 + ln (SAPS 3 score + 71.0599) ×13.2322 8.94 0.54 7.03 0.72 1.00 0.94–1.06 Central, Western Europe Logit=−36.0877 + ln (SAPS 3 score + 22.2655) ×7.9867 15.13 0.13 12.15 0.27 1.00 0.94–1.06 Eastern Europe Logit=−60.1771 + ln (SAPS 3 score + 51.4043) ×12.6847 10.13 0.43 7.12 0.71 1.00 0.92–1.08 North Europe Logit=−26.9065 + ln (SAPS 3 score + 5.5077) ×6.2746 3.45 0.97 2.22 0.99 1.00 0.86–1.14 Southern Europe, Mediterranean countries Logit=−23.8501 + ln (SAPS 3 score + 5.5708) ×5.5709 5.28 0.87 13.12 0.22 1.00 0.97–1.03 North America Logit=−18.8839 + ln (SAPS 3 score + 1) ×4.3979 4.22 0.93 4.47 0.92 1.00 0.86–1.14 GOF Ĥ GOF Ĉ p  O/E CI Discussion 17 18 19 Third, since computation of predicted mortality is based on a reference database, the user should be able to choose between them, i.e., a global database, which provides a broader comparison at the potential cost of less relevance to local conditions, and a regional database, which provides a better comparison with ICUs in geographic proximity but at the cost of losing comparability with ICUs in other parts of the world. A third possibility could be added—a country-representative database—but such a database would raise the problem of whether the ICUs selected were representative of a certain country. 20 21 22 23 25 24 26 4 27 28 29 30 Further studies should be done of factors occurring after ICU admission that influence risk-adjusted mortality. We should keep however in mind that this approach comes with one potential pitfall: a possible decrease in the amount of data available for the computation of the model; also, the shorter time period for data collection can eventually increase the likelihood of missing physiological data and the reliance on the assumption that missing physiological data are normal. This effect should be small, considering the widespread availability of monitoring and point-of-case analysers. 18 The SAPS 3 system was developed to be used free of charge by the scientific community; no proprietary information regarding the scientific content is retained. All the coefficients needed for the computation of outcome probabilities are available in the published material. The SAPS 3 can even be computed manually, using a simple scoresheet, although it was designed to be integrated into computerised data acquisition and storage systems that allow the automatic check of the quality of the registered data.  user-dependent problems 31  patient-dependent problems Users of these models should keep in mind that benchmarking is a process of comparing an ICU with a reference population. The appropriate choice of reference population is difficult, and we cannot simply change it because the observed-to-predicted mortality rate is not the one we want. For this reason, the choice should depend on the objective of the benchmark: more precise estimation will need local or regional equations, developed from a more homogeneous case mix. A generalisable estimation will, on the other hand, need more global equations developed from a more representative case mix. Last but not least, we have successfully addressed some of the problems of prognostic model development, especially those related to the underlying statistical assumptions for the use of specific methods for selection and weighting of variables and the conceptual development of outcome prediction models. In the future, multi-level modelling with varying slopes (and not just random intercepts) might be able to give a better answer to researchers but for the moment they would make the models to complex to be managed outside a research environment. Electronic Supplementary Material (PDF 794 KB)