Validation of prediction model for successful discontinuation of continuous renal replacement therapy: a multicenter cohort study

Article information

Kidney Res Clin Pract. 2024;43(4):528-537

Publication date (electronic) : 2024 June 20

doi : https://doi.org/10.23876/j.krcp.23.308

Junseok Jeon^1,*, Eun Jeong Ko^2,*, Hyejeong Park³, Song In Baeg⁴, Hyung Duk Kim⁵, Ji-Won Min², Eun Sil Koh⁶, Kyungho Lee¹, Danbee Kang³, Juhee Cho³, Jung Eun Lee¹, Wooseong Huh¹, Byung Ha Chung^,7,†, Hye Ryoun Jang^,1,†

¹Division of Nephrology, Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea

²Division of Nephrology, Department of Internal Medicine, The Catholic University of Korea, Bucheon St. Mary’s Hospital, Bucheon, Republic of Korea

³Center for Clinical Epidemiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea

⁴Division of Nephrology, Department of Internal Medicine, Myongji Hospital, Hanyang University Medical Center, Goyang, Republic of Korea

⁵Division of Nephrology, Department of Internal Medicine, The Catholic University of Korea, Eunpyeong St. Mary’s Hospital, Seoul, Republic of Korea

⁶Division of Nephrology, Department of Internal Medicine, The Catholic University of Korea, Yeouido St. Mary’s Hospital, Seoul, Republic of Korea

⁷Division of Nephrology, Department of Internal Medicine, The Catholic University of Korea, Seoul St. Mary’s Hospital, Seoul, Republic of Korea

Correspondence: Hye Ryoun Jang Division of Nephrology, Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-ro, Gangnam-gu, Seoul 06531, Republic of Korea. E-mail: shinehr@skku.edu

Byung Ha Chung Division of Nephrology, Department of Internal Medicine, Seoul St. Mary’s Hospital and Transplant Research Center, Convergent Research Consortium for Immunologic Disease, College of Medicine, The Catholic University of Korea, 222 Banpo-daero, Seocho-gu, Seoul 06591, Republic of Korea. E-mail: chungbh@catholic.ac.kr

*Junseok Jeon and Eun Jeong Ko contributed equally to this study as co-first authors.†Byung Ha Chung and Hye Ryoun Jang contributed equally to this study as co-corresponding authors.

Received 2023 November 23; Revised 2024 March 24; Accepted 2024 April 8.

Abstract

Introduction

Acute kidney injury (AKI) is a common condition among critically ill patients. A recent multicenter study reported that the incidence of AKI at all stages was 57% while that of stage 3 AKI was 30%; the mortality rate increased as AKI severity increased in critically ill patients [1]. Although continuous renal replacement therapy (CRRT) has not been proven superior to intermittent renal replacement therapy (RRT) in critically ill patients [2,3], CRRT is the preferred and mainly used RRT in critically ill patients because of its hemodynamic stability and high capacity to remove solutes or fluids. Meanwhile, CRRT has disadvantages including the risk of catheter-related complications, risk of bleeding associated with continuous anticoagulation, increased length of intensive care unit (ICU) stay, delayed rehabilitation, and removal of undesired substances such as antibiotics [4].

Therefore, it is important to discontinue CRRT for patients who no longer require it. However, consensus is lacking regarding the optimal timing of CRRT discontinuation. Urine output is reportedly the most consistent and important predictor of successful discontinuation of CRRT [5,6]. Some studies presented regression equations of urine output, serum creatinine, or sequential organ failure assessment score and showed excellent discriminative ability for successful discontinuation of CRRT [5,7,8]. However, no externally validated prediction models are simple or easy to apply.

We previously developed a prediction model derived from four clinical parameters for successful discontinuation of CRRT that was simple and showed good performance [9]. This study aimed to validate a prediction model for successful discontinuation of CRRT using temporal and external cohorts.

Methods

Results

Patients’ characteristics

The median patient age was 65 years (interquartile range, 54–76 years), and 61.8% were male (Table 1). The primary diagnoses upon admission were cancer (57.2%), heart failure (27.2%), and others (15.6%). Approximately 45% of patients (677 of 1,517) successfully discontinued CRRT. A higher proportion of patients had heart failure in the success versus failure group. Urine output was larger, and BUN and serum potassium levels were lower in the success versus failure group. However, MAP was comparable between the two groups. Age, proportion of primary diagnoses, and values of major variables on the day of CRRT discontinuation differed among institutions (Table 2). The rate of successful discontinuation of CRRT was 40% to 51%.

Table 1.

Characteristics of all patients of the pooled cohort

Table 2.

Patients’ characteristics in temporal and external validation set by cohorts

Performance of prediction model

The predicted and observed probabilities of successful discontinuation of CRRT in the entire cohort matched well according to each score (Fig. 1A) and the low- (0–2 points), intermediate- (3–5 points), and high-score (6–8 points) groups (Fig. 1B). The overall differences between the observed and predicted incidence were 3.0% (17.7% observed and 16.9% predicted probabilities), 3.6% (35.2% and 34.8%), and 2.0% (69.3% and 70.3%) in the low-, intermediate-, and high-score groups, respectively. The overall AUROC of the prediction model was 0.74 (95% confidence interval [CI], 0.71–0.76) (Fig. 1C). We assessed the AUROC according to the primary diagnosis at admission. Regardless of primary diagnosis, the prediction model showed good performance (AUROC [95% CI]: 0.76 [0.68–0.83] in heart failure, 0.75 [0.69–0.81] in cancer, and 0.73 [0.67–0.76] in others) (Fig. 2).

Figure 1.

Performance of prediction model for CRRT discontinuation.

The observed and predicted probabilities of successful discontinuation of CRRT were well-matched (A) in each score group and (B) in the low- (0–2), intermediate- (3–5), and high-score (6–8) groups. (C) The ROC curve of the prediction model.

AUROC, area under the curve of receiver operating characteristics; CI, confidence interval; CRRT, continuous renal replacement therapy; ROC, receiver operating characteristics.

Figure 2.

Performance of prediction model according to primary diagnosis at admission or cohorts.

The performance of the prediction model was good in all subgroups according to the primary diagnosis at admission.

AUROC, area under the curve of receiver operating characteristics; CI, confidence interval.

Performance of prediction model at each hospital

We further assessed the performance of the prediction model according to the institution. One temporal cohort and three external cohorts showed good discriminative ability (AUROC [95% CI]: 0.77 [0.73–0.81], 0.73 [0.69–0.77], 0.72 [0.63–0.82], and 0.73 [0.63–0.84]), but one external cohort (Hospital C) showed poor discriminative ability (0.56 [0.45–0.67]) (Fig. 3). This cohort, with poor discriminative ability, showed little association between the scores and the CRRT discontinuation success rate. The predicted and observed probabilities of successful discontinuation of CRRT in each cohort according to the score groups and scores are presented in Fig. 4 and Supplementary Fig. 1 (available online), respectively.

Figure 3.

Performance of prediction model according to each cohort.

The prediction model performed well in four cohorts but poorly in one cohort.

AUROC, area under the curve of receiver operating characteristics; CI, confidence interval; SMC, Samsung Medical Center.

Figure 4.

Observed and predicted probability of successful discontinuation of continuous renal replacement therapy according to prediction model low-, intermediate-, and high-score groups by cohort.

(A) Samsung Medical Center. (B) Hospital A. (C) Hospital B. (D) Hospital C. (E) Hospital D.

We analyzed each variable separately to elucidate what variable in the model caused the difference in performance between cohorts. Hospital C (AUROC 0.539 for urine output) showed very low discriminative ability for urine output compared to other hospitals (AUROC 0.698, 0.715, 0.665, and 0.683 for urine output in each hospital) (Supplementary Fig. 2, available online). Hospital C also tended to show lower or similar discriminative ability for BUN and serum potassium, but higher discriminative ability for MAP compared to other hospitals (Supplementary Figs. 3–5, available online).

Differences in the strategies of continuous renal replacement therapy operation

We compared differences in the overall steps of CRRT operation to elucidate the cause of the differences in model performance between hospitals. Table 3 shows CRRT operation strategies regarding nephrology engagement and CRRT-dedicated nursing staff of each hospital. In Samsung Medical Center and the three hospitals where the performance of the prediction model was good, nephrologists determined all important steps of CRRT operation including initiation, prescription with monitoring the adequacy of CRRT, and screening the possibility or final decision of CRRT discontinuation by multidisciplinary care with critical care physicians. On the other hand, both prescription and discontinuation of CRRT were primarily controlled by attending physicians in the Hospital C although only initiation of CRRT was determined by nephrologists under on-demand consultation.

Table 3.

The strategies of CRRT operation for each hospital

Discussion

Here we validated our simple prediction model for successful discontinuation of CRRT. The prediction model showed good discriminating ability for one temporal cohort and three external cohorts but showed poor performance for one external cohort. Our prediction model was based on urine output, MAP, BUN, and serum potassium levels. This is simple and easy to apply in clinical practice; however, as shown in one external cohort, its usefulness can be affected by the strategies of CRRT operation.

Consensus is currently lacking on CRRT discontinuation criteria. The prolonged maintenance of CRRT increases the length of ICU stay and overall costs, delays mobility, and exposes patients to CRRT-associated complications [4]. On the other hand, whether reinitiating RRT per se after CRRT discontinuation adversely affects mortality or renal outcomes has not been sufficiently studied. Previous observational studies demonstrated that 40%–50% of patients with AKI requiring RRT failed to discontinue it and had higher mortality rates and a lower chance of renal recovery than those who successfully discontinued RRT [6,10–12]. These differences may reflect the overall disease severity of patients rather than the adverse effects of RRT reinitiation. However, if kidney function does not sufficiently recover after CRRT is discontinued, the patient is at risk of recurrent volume overload, uremia, and electrolyte imbalance. In addition, approximately 50% of patients experience intradialytic hypotension when CRRT is switched to intermittent RRT, which can cause additional ischemic injury and aggravate organ dysfunction [13]. Therefore, the appropriate timing of CRRT discontinuation is important.

Urine output was the only common significant factor associated with successful discontinuation of CRRT across several previous studies and in our study [5–9,14]. Although most previous studies were small-scale, the largest multicenter study reported urine output, increased urine output, serum creatinine, history of chronic kidney disease, and CRRT duration as significant factors for successful discontinuation of CRRT [6]. Two previous studies presented calculation formulas using a regression model [5,7]. However, despite their high AUROC, they did not provide a simplified scoring system; thus, these models could be less practical for clinical decisions. One study presented a simple scoring system and a high AUROC with kinetic estimated glomerular filtration rate and urine output; however, kinetic estimated glomerular filtration rate itself requires serial serum creatinine values and a complex calculation formula [8]. In addition, the prediction models reported in previous studies have not been validated. Our prediction model, which is based on a simple scoring system with clinical parameters that can be easily obtained daily, has a relatively good AUROC value and good discriminative ability [9]. Furthermore, we validated the model in both temporal and external cohorts.

Four clinical variables were selected for our model and applied regardless of the patient’s medical condition or diagnosis. However, the relative importance of each variable may vary depending on the patient’s underlying disease or clinical setting. Although the performance of our simple model did not depend on primary diagnoses, more accurate and differential weighting of each variable according to the patient’s condition or clinical setting may improve the universality and optimality of the model. On the other hand, complex model may be less clinically useful. Further research is needed to improve the performance and universality of our model with simple clinical variables.

It is worth noting that the performance of our prediction model was good in the four cohorts but poor in one external cohort. This discrepancy can be caused by differences in CRRT application, operation policies, and patient characteristics. Only one institution demonstrated poor model performance, whereas the other two institutions with similar numbers of patients demonstrated good model performance. In addition, urine output, which has been reported as an important factor predicting CRRT discontinuation or kidney recovery in most previous studies [5–9,14], was not helpful in predicting successful discontinuation of CRRT in one institution with poor model performance. Therefore, poor performance of prediction model in one institution seems to be attributed to CRRT operation strategy rather than patient characteristics. In one institution showing poor performance of the model, the nephrology staffs or team played only a limited role in deciding the initiation of CRRT as consultants and did not actively engage to control and discontinue CRRT. Therefore, our study strongly supports the necessity of active intervention by nephrology team or specialized CRRT team for optimal CRRT management and good performance of our prediction model. On the other hand, our prediction model may also help clinicians make appropriate decisions regarding CRRT discontinuation even in institutions without active participation of nephrology staffs. Further prospective studies are needed to verify whether our prediction model can effectively assist the adequate discontinuation of CRRT in institutions with limited nephrological resources.

In South Korea, switching CRRT to intermittent HD is actively attempted in patients whose overall hemodynamic status has been relatively stabilized even during ICU stay, and this effort for the transition of the RRT modality has been driven by the Korean National Health Insurance Service. This policy makes a significant portion of CRRT discontinuation in South Korea aimed at transitioning to intermittent HD before sufficient renal recovery, rather than complete discontinuation of RRT. Therefore, our findings should be interpreted with caution in countries or institutions whose primary goal of CRRT discontinuation is to discontinue RRT completely. Nevertheless, our prediction model is still useful when attempting to use a scoring system to aid clinical decisions regarding transitioning from CRRT to intermittent HD or maintaining CRRT.

This study had several limitations. First, it was a retrospective cohort study. However, several cohorts were used to validate the prediction model, which may minimize or overcome potential bias. Second, our multicenter study included only Korean ICU patients covered by the Korean National Health Insurance Service. Therefore, our prediction model requires validation in other races or countries. Third, our prediction model showed poor performance in one external cohort, suggesting the necessity of active intervention by nephrology team supporting the entire process as a prerequisite for applying our prediction model. However, as mentioned above, if our prediction model is applied appropriately in an institution with limited involvement of nephrology team, it has the potential to help decisions regarding CRRT discontinuation. Fourth, detailed causes of AKI requiring RRT were not included in the analyses. The retrospective data collection inevitably limits the accurate capture of such data. To overcome this limitation, we conducted a subgroup analysis based on the primary diagnosis at admission and demonstrated that the model was effective for each group.

In conclusion, the good performance of our prediction model for successful discontinuation of CRRT was validated in all four cohorts, except for one cohort in which nephrologists did not actively engage in CRRT operation. This model can help discriminate between patients with a high or low probability of successful discontinuation of CRRT depending on the active engagement of nephrologists in CRRT operation. Further prospective multicenter studies with protocolized CRRT management considering the active engagement of nephrologists in CRRT operation are required to validate and improve our predictive model.

Supplementary Materials

Article information Continued