Predicting suicidal behaviour without asking about suicidal ideation Machine Learning and the role of Borderline Personality Disorder criteria

Running head: PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING Predicting suicidal behaviour without asking about suicidal ideation: Machine Learning and the role of Borderline Personality Disorder criteria Adam Horvath, Mark Dras, Catie C.W Lai, Simon Boag Macquarie University Author Note Adam Horvath, Department of Psychology, Macquarie University; Mark Dras, Department of Computing, Macquarie University; Catie C.W Lai, Department of Psychology, Macquarie University; Simon Boag, Department of Psychology, Macquarie University Correspondence concerning this article should be addressed to Adam Horvath, Department of Psychology, Macquarie University, New South Wales, 2019 E­mail: adam.horvath@mq.edu.au PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING Abstract Identifying factors that predict who may be at risk of suicide could help prevent suicides via targeted interventions It is difficult at present, however, to predict which individuals are likely to attempt suicide, even in high­risk populations such as Borderline Personality Disorder (BPD) sufferers The complexity of person­situation dynamics means that relying on known risk factors may not yield accurate enough results for prevention strategies to be successful Furthermore, risk models typically rely on suicidal thoughts, even though it has been shown that people often intentionally withhold this information To address these challenges, this study compared the performance of six machine learning and categorisation models in terms of accurately identifying suicidal behaviour in a prison population (n = 353), by including or excluding questions about previous suicide attempts and suicidal ideation Results revealed that modern machine learning algorithms, especially gradient tree boosting (AUC = 875, F1 = 846), can accurately identify individuals with suicidal behaviour, even without relying on questions about suicidal thoughts, and this accuracy can be maintained with as low as 29 risk factors Additionally, based on this evidence, it may be possible to implement a decision tree model using known predictors to assess individuals at risk of suicide These findings highlight that modern classification algorithms not necessarily require information about suicidal thoughts for modelling suicide and self­harm behaviour Keywords: suicide prevention, borderline personality disorder, machine learning, prediction, classification, bpd, tree boosting PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING Predicting suicidal behaviour without asking about suicidal ideation: Machine Learning and the role of Borderline Personality Disorder criteria Suicide is a major global health issue, with 800,000 deaths by suicide each year Additionally, for each suicide, there are an estimated 20 or more suicide attempts (World Health Organization, 2014) Identifying factors that predict who may be at risk of suicide could help prevent suicides via targeted interventions and so reduce more deaths (Mann et al., 2005) There is broad agreement, however, that it is particularly challenging to identify who will die by suicide (Franklin et al., 2017; Pestian et al., 2017) Among these challenges are methodological limitations, which have prevented testing the complex interaction of factors associated with suicide risk (Franklin et al., 2017) Regardless, some populations are known to be more at risk than others Borderline personality disorder (BPD), for instance, is especially associated with elevated suicide risk (Chesney et al., 2014) and has an estimated community prevalence of around 1–2% (Black et al., 2004; Gunderson et al., 2013) BPD is characterised by instability of self­image, interpersonal relationships and affects, accompanied by impulsivity, risk­taking, and potential hostility and suicidal ideation (American Psychiatric Association, 2013) Compared to the rest of the population, deliberate self­harm (69–80%), suicide attempts (75%), and completed suicide rate (10%) are much higher in persons with BPD (Black et al., 2004; Brown & Chapman, 2007) Suicide attempt rates are especially high when BPD sufferers are in their twenties (American Psychiatric Association, 2006), while the suicide completion rate is highest for individuals in their thirties (Biskin, 2015) Presently, however, it is difficult to predict which individuals are likely to attempt suicide, even in high­risk populations such as BPD sufferers There are known risk factors for suicide which, in theory, could help with prediction Feelings of helplessness, sadness, anxiety, and negative affect, for instance, are known to be associated with suicidality (Podlogar et al., 2018) Regarding BPD, crisis­generating behaviour, interpersonal conflict (Brown & Chapman, 2007), and stressors resulting from past impulsive or avoidant behaviours (e.g., accruing a large debt) are also known risk factors (Brown & Chapman, 2007) Merely relying on known risk factors, however, may not yield accurate enough results for prevention strategies to be successful (Franklin et al., 2017) For PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING example, even seemingly important cues, such as expressed suicidal ideation, can have low predictive validity: only around 40% of people who die by suicide express suicidal thoughts at an earlier time (McHugh et al., 2019) This may be partly due to factors such as impulsive, unplanned suicide attempts, or concerns regarding the outcome of disclosure (e.g., stigma, and being hospitalised or medicated) (Richards et al., 2019) In the latter case, evidence indicates that certain populations are reluctant to disclose such information For example, older people (Heisel et al., 2010), some cultures (Takeuchi & Nakao, 2013), outpatients (Earle et al., 1994), and children and adolescents (Bolger et al., 1989) are all known to withhold information about suicidal ideation Suicide questionnaires may consequently fail to identify the majority of people who would attempt suicide shortly after answering the questions (Richards et al., 2019) Furthermore, health professionals may feel uncomfortable asking about suicidal ideation, fearing that such questions might increase the likelihood of a suicide attempt — even though this does not appear to be the case (Bajaj et al., 2008; Stoppe et al., 1999) As such, suicide prevention strategies that could accurately identify at­risk individuals without relying on reported suicidal ideation may have practical advantages Franklin et al (2017) state that prediction accuracy has not improved significantly over the past 50 years For example, suicide and self­harm prediction models employing logistic regression tend to reach AUC scores of only 6–.7 (e.g., Horton, 2018; Kessler et al., 2017; Walsh et al., 2017) An AUC score can be interpreted as the probability that any randomly selected suicidal subject would receive a higher prediction score than any other randomly selected non­suicidal subject (Fawcett, 2006) As such, an AUC score of indicates a random classification performance, and models producing scores of 6–.7 generally provide only poor levels of discrimination Furthermore, around 80% of studies investigating the predictability of suicide rely on only five broad risk factors for predicting suicidal behaviour, and the typical accuracy of these algorithms is only slightly better than chance (Franklin et al., 2017) This suggests overall that simple models built on relatively few risk factors, and not accounting for complex person­situation dynamics, cannot provide high enough accuracy to target individuals at risk These findings have led some researchers to believe it is not possible to predict suicide PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING attempts (Black et al., 2004) However, such conclusions are based on algorithms, discussed above, which are not suitable for addressing the complex non­linear interactions between a large number of predictors Although algorithms identifying suicidal behaviour using relatively few predictors may not yield practical results, more recent attempts using machine learning (ML) have demonstrated much higher prediction accuracy (Burke et al., 2019) Walsh et al (2017), for instance, achieved relatively high suicide prediction accuracy using ML models based on health records of adults with deliberate self­harm (AUC = 84) Jung et al (2019) also achieved similar accuracy (AUC = 86) using gradient tree boosting modelling, identifying adolescents with suicidal ideation and suicidal behaviour Unlike conventional statistical models, such ML methods can be applied even where there are a large number of predictors relative to the sample size ML methods in these circumstances avoid overfitting the training data (i.e., avoids learning a set of model parameters that may give zero error on the training data but does not generalise to other data, through a range of techniques; Hastie et al., 2001) For example, regularisation, which is frequently used in ML models, penalises model complexity; cross­validation, or testing the model’s accuracy on a held­out dataset, allows an estimate of the average generalisation error when the method is applied to a new, independent test sample Hastie et al (2001, pp 247­9) showed that these approaches yield accurate models even for a large number of predictors relative to sample size Given the complexity of person­situation dynamics, accurate prediction of suicidal thoughts and behaviours may need to take into account hundreds of risk factors (Franklin et al., 2017) It is also possible, however, that after the aforementioned regularisation, an accurate prediction would only require a much smaller subset of risk factors For instance, Ribeiro et al (2019) achieved good accuracy predicting suicidal ideation and attempts (AUC = 83 ­.89) using random forest with 51 variables This finding demonstrates that ML methods can accurately predict suicide­related behaviours using a relatively small number of risk factors Nevertheless, to translate these risk algorithms into clinical practice, the predictors need to be reduced to a more manageable and practical amount Reducing the number of predictors is also beneficial with respect to interpretation Even PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING though ML techniques may potentially achieve high enough accuracy to target individuals with suicidal behaviour (Walsh et al., 2017), they often work as a black box This makes it difficult to understand what risk factors can predict suicide attempts, which means these models can then be prohibitively complex to interpret Recent ML research has started to address this by focusing on both the interpretability and visualisation of these complex models, regaining some insights on how the combination of predictors contributed to the output of a model (Lundberg & Lee, 2017; Tan et al., 2018) As such, there are avenues available for addressing this potential limitation Some of these easy­to­interpret models provide a much clearer, simpler indication of how various parts of the complete model operate These include decision trees, which can provide a compromise between the complexity and accuracy of the ML models, and the interpretability and simplicity of the more traditional models (Quinlan, 1987) Decision tree­based modelling involves creating a tree­like structure that represents questions and answers, and their consequences, such as the chance of any given outcome Recent research focusing on the prediction of suicidal ideation and suicidal behaviour has been able to build decision tree models with potential clinical significance (e.g., Batterham & Christensen, 2012; Handley et al., 2014; Mann et al., 2008) For instance, the decision tree model by Handley et al (2014) accurately predicted (AUC = 81) suicidal ideation in older adults on a five­year follow­up study Consequently, decision trees might provide a viable middle ground between modern ML algorithms and simpler models for successfully predicting suicidal behaviour Given that results to date using simple models indicate generally poor predictive validity of suicidal behaviour (e.g., Horton et al., 2018; Kessler et al., 2017), and that modern ML algorithms show promising potential in identifying individuals at risk of suicide (Burke et al., 2019), the present study compared the performance of different ML and categorisation models in terms of predicting suicide attempts ML models studied include random forest and tree boosting, which typically yield better prediction than traditional algorithms such as logistic regression (Couronné et al., 2018; Neumann et al., 2004) We also tested the accuracy of neural network models which, in some cases, outperform even random forest models (Jaimes et al., 2005; Raczko & Zagajewski, 2017; Were et al., 2015) The present study applied these PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING ML algorithms to retrospective modelling based on prison data to determine whether suicidal behaviour could be successfully predicted Previous findings indicate that retrospective and predictive modelling of suicidal behaviour achieve similar accuracy (Walsh et al., 2017), which suggests that the same datasets and models can be used for both predictive and retrospective modelling of suicidal behaviour, addressing a potential limitation of our retrospective modelling Prisons tend to have a higher suicide rate than the rest of the population (Naud & Daigle, 2013), and prisoners have exceptionally high rates of personality disorders, including antisocial personality disorder (APD) and BPD (Fazel & Danesh, 2002) Some researchers argue that these disorders are merely different representations of the same underlying psychopathology (Paris, 1997), and further studies show that APD and BPD share at least some behavioural and neurobiological background (Black et al., 2010; Buchheim et al., 2013) As such, ML models could potentially help both with building accurate suicide prediction models, as well as identifying how BPD and APD contribute to the risk of suicide attempts In summary, the present study sought to improve on existing ML models by comparing the performance of six machine learning and categorisation models in terms of accurately identifying suicidal behaviour in a prison population (n = 353), both relying on questions about previous suicide attempts and suicidal ideation, and not Despite the recent advancements in ML models for testing prediction of suicide­related behaviours, for translation into clinical practice, there is a need for risk algorithms/ML models that use a relatively small number of predictors There are also limitations with including suicidal ideation as a predictor, since around 60% of people who die by suicide not express suicidal ideation at an earlier time (McHugh et al., 2019), and some populations are known to withhold this information, as discussed above Accordingly, it is important to test the models’ accuracy after excluding suicidal ideation We hypothesised that modern ML algorithms, especially random forest and gradient tree boosting, could accurately identify individuals with suicidal behaviour within prison population even after excluding questions about suicidality, given enough personal details such as factors related to demographics, physical and mental health, substance abuse, and criminal history We further expected that BPD diagnostic criterion PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING would be an important predictor in these models We were also interested in how APD diagnostic criteria would contribute to the model, given the similarities between the disorders Finally, in order to help translate risk algorithms into clinical practice, we wanted to investigate whether a smaller number of predictors could be used to predict suicide attempts To allow us to better understand how many personal variables are required to build accurate risk models, we were interested in how reducing the number of input variables would affect the models’ performance Method Materials For this exploratory study, the Inter­university Consortium for Political and Social Research catalogue was searched for datasets that contained detailed BPD diagnostic data and a large number of participants (n > 100) to build ML models on In the identified dataset (Sacks & Melnick, 2011), one dependent variable, ‘suicide ever attempted’ as reported by the participants, and 915 independent variables were identified These were reduced to 641 after removing any suicide­related and non­numeric variables Some of the variables in the dataset, such as the Mental Health Screening Form Total Score and BPD diagnosis, were derived from other variables using simple formulas, such as if­else and addition (e.g., bpddiag = pdborder, 0through4 = 0, ELSE = 1) We kept both the source variables and the derived predictors in the dataset Participants The dataset described US prisoners who were participating in prison­based substance abuse treatment programs across 14 facilities Table shows high­level demographics and mental health data about individuals, and how the total dataset was randomly split into a training dataset and a validation dataset Data about income, education, and socioeconomic status was not available from the participants PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING Table Descriptive statistics Dataset Gender Male Female Ethnicity White Latino African American Suicide attempt Yes No BPD BPD 0­4 BPD 5­9 Suicide attempt Yes No Gender Male Female APD APD 0­2 APD 3­7 Suicide attempt Yes No Gender Male Female Total Total Training Validation 207 (58.64) 146 (41.36) 163 (57.80) 119 (42.20) 44 (61.97) 27 (38.03) 137 (38.81) 120 (33.99) 96 (27.20) 109 (38.65) 97 (34.40) 76 (26.95) 28 (39.44) 23 (32.39) 20 (28.17) 59 (16.71) 294 (83.29) 45 (15.96) 237 (84.04) 14 (19.72) 57 (80.28) 303 (85.84) 50 (14.16) 246 (87.23) 36 (12.77) 57 (80.28) 14 (19.72) 27 (54.00) 23 (46.00) 18 (50.00) 18 (50.00) (64.29) (35.71) 18 (36.00) 32 (64.00) 13 (36.11) 23 (63.89) (35.71) (64.29) 199 (56.37) 154 (43.63) 164 (58.16) 118 (41.84) 35 (49.30) 36 (50.70) 32 (20.78) 122 (79.22) 23 (19.49) 95 (80.51) (25.00) 27 (75.00) 103 (66.88) 51 (33.12) 78 (66.10) 40 (33.90) 25 (69.44) 11 (30.56) 353 (100) 282 (79.89) 71 (20.11) Note Items are raw counts, percentages in parentheses APD = antisocial personality disorder, diagnostic criteria ≥ 3; BPD = borderline personality disorder, diagnostic criteria ≥ PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 10 Procedure We built six predictive models in Python programming language to compare their accuracy, both against each other and to other published results from similar studies We wanted to compare different modelling approaches with different bias­variance tradeoffs (Hastie et al., 2001); in other words, models that can infer simple rules but potentially underfit the data (high bias, low variance, such as generalised linear models), and models that can infer arbitrarily complex rules but might overfit the data (low bias, high variance, such as tree­based models) The six models were: gradient tree boosting, implemented in the Xgboost toolkit (Chen & Guestrin, 2016); a fully connected three­layer neural network (multilayer perceptron), implemented in Keras on top of Tensorflow (Chollet et al., 2015); random forest; decision tree; logistic regression; and linear regression with a simple cutoff classifier (Pedregosa et al., 2011) The dataset was split into a training and validation dataset, the latter of which was used to verify the generalisability of the models after training Due to the small sample size, the validation dataset was also used to parameter tune the neural network model The neural network model was trained on normalised predictors as this model is known to not learn well on raw input scores, especially when some of the scores have a special meaning (e.g., ­9: missing data) The rest of the models were trained on the raw dataset Analysis of the models We planned to evaluate the models according to several different criteria We included AUC because it is the most typically reported measure in literature AUC, the total area under the Receiver Operating Characteristic (ROC), shows the relationship between sensitivity (true positive rate) and specificity (true negative rate) at different cutoff values On a balanced dataset, where the number of positive and negative samples are roughly equal, a random classifier would yield an AUC of 5, and a perfect classifier would yield 1.0 However, we note a major limitation of AUC, namely that it is misleading in the case of imbalanced classes, such as the ratio of suicide­attempters to non­attempters (Raeder et al., 2012) We also included positive predictive value (PPV) and sensitivity, so that our findings can be easily compared to other papers’ results (Belsher et al., 2019) Finally, we also reported the F1 scores, the harmonic mean of precision and recall, which is a standard measure in ML F1 scores focus purely on the positive class of interest, not 14 Overall risk PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING BPD diagnostic criteria met Figure SHAP dependency plot of the number of BPD diagnostic criteria met The vertical dispersion of the dots indicate interaction with other variables in the dataset A broader dispersion means more significant interactions Higher overall risk values indicate a higher risk of a suicide attempt from the topmost rectangle, then following up with the left question if the answer was true, or the right if the answer was false, repeating this process until reaching a predicted risk value The recommended cutoff was 667 Further details of specific variables can be found in the publication of Sacks and Melnick (2011) PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING Figure The decision tree model, showing the predicted value (v) for suicidal behaviour This model can be administered by asking the question 15 PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 16 Model reduction One of the frequent criticisms about ML models is that they use too many input parameters to achieve their accuracy, sometimes skewing the input variable­to­data ratio To address this, we evaluated the accuracy of our tree boosting model, which was trained without suicidal ideation, by reducing the number of input parameters After inspecting all the initial non­zero parameter weights in the SHAP model, we identified that the model relied on only 39 input parameters out of the available 641 Not using all the input variables is expected, as the model uses regularisation, but we did not anticipate such a low number of parameters As Franklin et al (2017) noted, ML models might need to include hundreds of variables, which was not true in our analysis We ordered the input parameters by their importance, as weighted by their SHAP score, and removed them one by one — starting with the least important feature — until reaching only one parameter in the model (see Figure for the accuracy of the model during model reduction) Figure Accuracy of the tree boosting model, without suicidal ideation, during model reduction The highest F1 was achievable with as little as 29 variables (highlighted) To our surprise, the tree boosting model, even without taking suicidal ideation into account, maintained the same F1 score as the full model with as little as 29 input parameters This means that including the right parameters matters most when it comes to predicting suicidal behaviour, rather than simply increasing the number of predictors PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 17 Discussion The aim of this study was to compare the performance of different ML and classification algorithms both in terms of accurately identifying suicidal behaviour using a wide variety of predictors, including BPD criteria, and comparing how including or excluding data about suicidal ideation would affect the outcome A further aim was to determine whether a smaller number of predictors could be used to predict suicide attempts, in order to help translate risk algorithms into clinical practice We hypothesised that modern ML algorithms could accurately identify suicidal behaviour based on various individual factors, including demographics, physical and mental health, substance abuse, and criminal history data, even without including any suicide­related questions in the model In line with previous research, no unique set of predictors emerged in the models that could identify suicidal behaviour Instead, our findings indicate that it appears possible to identify suicidal behaviour with high accuracy, starting with a large number of predictors and reducing the model to a small, relatively easy­to­administer set of questions Furthermore, meeting BPD diagnosis is an important risk factor within these models APD on the other hand, in the presence of other predictors, did not emerge as an important variable in the models Overall, it might be possible to predict suicide risk using only relatively few predictors and meeting a certain number of BPD criteria, even within non­BPD populations Regarding risk factors, our most accurate machine learning model was based on tree boosting, and it determined that meeting five or more diagnostic criteria of BPD evenly increases the risk of suicide Furthermore, in line with previous research, our models indicated that previous psychiatric hospitalisation is an important risk factor when predicting suicidal behaviour (e.g., Franklin et al., 2017; Kessler et al., 2017) Our findings suggest overall that successfully predicting suicidal behaviour requires addressing complex interactions of many factors, but – and unlike previous research – the models not need to rely on hundreds of risk factors Instead, the interaction of a smaller number of predictors is important, and gradient tree boosting automatically includes these meaningful interactions in the model (Dietterich et al., 2004) The small number of variables is potentially important at a clinical level, since efficient and effective assessment allows for quicker identification of individuals PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 18 at risk of suicide and more rapid intervention while minimising non­compliance This may be particularly important with BPD patients, where volatile emotional responses demand both a practical and efficient approach (Hong, 2016) Regarding the accuracy of our models, the logistic regression model produced a similar AUC score to models trained on different populations, such as with Horton (2018), Kessler et al (2017), and Walsh et al (2017), who reported 671, [0.67, 0.72], and 66, respectively This reinforces the view that logistic regression should not be used for modelling suicidal behaviour because of its low accuracy Additionally, in line with previous research that compared the general performance of classification algorithms, we found that random forests outperform logistic regression models (Couronné et al., 2018) It is also noteworthy, however, that the random forest model, based on the relatively small sample size of our dataset, achieved a similar AUC to Walsh et al (2017), who reported a value of 84 Walsh et al.’s model was built on a much larger dataset (n = 5,543), and so our findings suggest that smaller sample sizes might be adequate for building predictive models for suicidal behaviour, and that what matters most when it comes to model accuracy is the type of input parameters, not the number of them To our surprise, though, the neural network model did not perform as well as expected, even after attempting to finetune the model parameters, including the number and size of the layers, learning rate, number of epochs, regularisations, dropout ratios, and class weights This may be because our chosen neural network architecture would typically require much larger training samples (Chan et al., 1999), and neural network models are generally better suited for inputs that are on a continuous scale rather than categorical values Even though decision trees are underutilised for diagnostic purposes, our decision tree model yielded reasonably accurate results compared to our most accurate ML model One advantage of decision trees is that they are straightforward to administer using either electronic or pen­and­paper methods Our results indicate that our decision tree might be used for pre­screening or diagnostics in time­pressured situations (see Figure 2) For instance, the identified predictors could serve as a guided interview for triaging individuals at risk of suicide The decision tree could also be used without asking about past suicidal behaviour or suicidal ideation This has clear benefits since, as noted earlier, most of the people who die by PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 19 suicide not express suicidal ideation at an earlier time (McHugh et al., 2019; Richards et al., 2019) Furthermore, some populations are reluctant to disclose suicidal ideation and doctors may be reluctant to ask about it (Bajaj et al., 2008; Stoppe et al., 1999) This approach could therefore potentially help with targeted early intervention, which is especially important in the case of BPD sufferers, who frequently present at psychiatric and medical care settings (Gunderson et al., 2013; Hong, 2016) Strength, limitations, and future directions Future research could investigate the accuracy and usefulness of the decision tree, and whether it would be possible to build more targeted diagnostic questionnaires from the existing items using feature engineering Feature engineering is a process of creating more meaningful variables that the ML models can utilise, either based on theory or automatically, by arbitrarily combining attributes and checking the resulting model’s accuracy using the new attributes (Khurana et al., 2016) Currently, feature engineering is exceptionally computationally intensive if the number of variables is large However, given that our findings use relatively few predictors, feature engineering is a possible future direction for this research A major strength of this study is that it is the first study to compare the performance of modern machine learning models in predicting suicidal behaviour, both with and without including suicidal ideation as a variable, concluding that suicidal ideation is not required to build accurate suicidal behaviour classification models Furthermore, to our knowledge, this is the first study to demonstrate that gaining insights into these ML models can highlight how individual risk factors, such as the number of BPD diagnostic criteria met, contribute to the overall risk of suicidal behaviour, as well as how reduced models, with much fewer input variables, can achieve the same performance as a model trained on hundreds of input parameters Despite the apparent successful modelling reported here, however, there are a number of limitations in the present study First, the ML predictions were based on retrospective, unsuccessful suicide attempts, as reported by the participants, and so it remains to be seen if suicide attempts can be accurately predicted before they occur It should also be kept in mind PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 20 that, while we treated false­positive results from our models as truly incorrect classifications, it is also possible that those individuals who reported not having attempted suicide may attempt suicide in the future As such, these cases would not be false negatives at all, but instead candidates for interventions Further research could investigate this, and the modelling needs to be repeated and the accuracy redetermined to assess longer­term predictability Furthermore, the prison psychiatric sample data is not representative of the general population Incarceration is associated with increased suicide risk (Naud & Daigle, 2013), raising questions about the generalisability of our findings to non­prison populations Nevertheless, our results are comparable to findings from studies using non­prison populations (e.g., Walsh et al., 2017), although training our models on non­prison populations is an obvious direction for future research At the same time, BPD suicidality in our dataset (54%) was lower than reported in previous studies (e.g., Black et al., 2004; Brown & Chapman, 2007), which might be partially explained by the selectivity of the subjects into the program, and by the missing data on suicide completers It is also possible, given the US dataset, that the same ML models may not be predictive cross­culturally Forecasting suicidal behaviour based on meeting diagnostic criteria for BPD might be particularly problematic if the scores can vary across cultures (Wang et al., 2012), especially given that cultural acceptance of suicide differs around the world (Gunderson et al., 2013) The dataset also had a small sample size in terms of machine learning, which optimally requires thousands of data points However, as noted earlier, we achieved comparable results to previous studies on larger sample sizes (e.g., Walsh et al., 2017) This possibly suggests that the quality of the predictors is more important than having a large sample size, and that the models identifying suicidal behaviour may not directly benefit from larger samples Nevertheless, a larger sample size would help with validating the robustness of our models Finally, we did not have the age of the participants or when they attempted suicide even though, as noted earlier, age is known to be an important risk factor in suicidality (American Psychiatric Association, 2006; Biskin, 2015) Data on the date of attempts and the current age of the person may increase the accuracy of the models PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 21 Conclusion The aim of this study was to compare the performance of different classification and ML models in terms of accurately identifying suicide attempts using a wide variety of predictors, including BPD criteria, and showing how including or excluding questions about past attempts or suicidal ideation would affect the outcome A further aim was to determine whether a smaller number of predictors could be used to predict suicide attempts, in order to help translate risk algorithms into clinical practice The findings indicate that it appears to be possible to predict suicide attempts with high accuracy based on a relatively small number of predictors, even without relying on information about previous attempts or suicidal thoughts Consequently, modern ML models, especially tree boosting, appear to be able to accurately identify individuals at risk of suicide Additionally, based on this evidence, it may be possible to implement a decision tree model using known predictors to assess individuals at risk of suicide Although these findings are based on retrospective data, they suggest a promising future in identifying individuals at risk of suicide and potentially preventing suicide from occurring Authorship A.H developed the study concept, performed the data analysis, drafted the paper, contributed to the conceptualisation of the study and interpretation of the data analysis, and provided critical revisions M.D contributed to the data analysis and its interpretation, and provided critical feedback and revisions C.L contributed to the conceptualisation of the study and interpretation of the data analysis, and provided critical feedback and revisions S.B drafted the paper, contributed to the conceptualisation of the study and interpretation of the data analysis, provided critical feedback and revisions, and supervised the project All authors approved the final version of the paper for submission Supplemental Material Interested readers may contact the corresponding author for the decision tree with the original variable names PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 22 References American Psychiatric Association (2006) Practice Guideline for the Treatment of Patients With Borderline Personality Disorder In APA Practice Guidelines for the Treatment of Psychiatric Disorders: Comprehensive Guidelines and Guideline Watches (1st ed.) Arlington, VA, American Psychiatric Association American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders: DSM­5 (5th ed) Arlington, VA, American Psychiatric Association Bajaj, P., Borreani, E., Ghosh, P., Methuen, C., Patel, M., & Joseph, M (2008) Screening for suicidal thoughts in primary care: The views of patients and general practitioners Mental Health in Family Medicine, 5(4)pmid 22477874, 229–235 Batterham, P J., & Christensen, H (2012) Longitudinal risk profiling for suicidal thoughts and behaviours in a community cohort using decision trees Journal of Affective Disorders, 142(1­3), 306–314 https://doi.org/10.1016/j.jad.2012.05.021 Belsher, B E., Smolenski, D J., Pruitt, L D., Bush, N E., Beech, E H., Workman, D E., Morgan, R L., Evatt, D P., Tucker, J., & Skopp, N A (2019) Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation JAMA Psychiatry, 76(6), 642 https://doi.org/10.1001/jamapsychiatry.2019.0174 Biskin, R S (2015) The Lifetime Course of Borderline Personality Disorder The Canadian Journal of Psychiatry, 60(7), 303–308 https://doi.org/10.1177/070674371506000702 Black, D W., Blum, N., Pfohl, B., & Hale, N (2004) Suicidal Behavior in Borderline Personality Disorder: Prevalence, Risk Factors, Prediction, and Prevention Journal of Personality Disorders, 18(3), 226–239 https://doi.org/10.1521/pedi.18.3.226.35445 Black, D W., Gunter, T., Loveless, P., Allen, J., & Sieleni, B (2010) Antisocial personality disorder in incarcerated offenders: Psychiatric comorbidity and quality of life Annals of Clinical Psychiatry: Official Journal of the American Academy of Clinical Psychiatrists, 22(2)pmid 20445838, 113–120 Bolger, N., Downey, G., Walker, E., & Steininger, P (1989) The onset of suicidal ideation in childhood and adolescence Journal of Youth and Adolescence, 18(2), 175–190 https://doi.org/10.1007/BF02138799 PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 23 Brown, M Z., & Chapman, A L (2007) Stopping Self­Harm Once and for All: Relapse Prevention in Dialectical Behavior Therapy In Therapist’s Guide to Evidence­Based Relapse Prevention (pp 191–213) Elsevier https://doi.org/10.1016/B978­012369429­4/50039­2 Buchheim, A., Roth, G., Schiepek, G., Pogarell, O., & Karch, S (2013) Neurobiology of borderline personality disorder (BPD) and antisocial personality disorder (APD) Schweizer Archiv für Neurologie und Psychiatrie, 164(4), 115–122 https://doi.org/10.4414/sanp.2013.00156 Burke, T A., Ammerman, B A., & Jacobucci, R (2019) The use of machine learning in the study of suicidal and non­suicidal self­injurious thoughts and behaviors: A systematic review Journal of Affective Disorders, 245, 869–884 https://doi.org/10.1016/j.jad.2018.11.073 Chan, H.­P., Sahiner, B., Wagner, R F., & Petrick, N (1999) Classifier design for computer­aided diagnosis: Effects of finite sample size on the mean performance of classical and neural network classifiers Medical Physics, 26(12), 2654–2668 https://doi.org/10.1118/1.598805 Chen, T., & Guestrin, C (2016) XGBoost: A Scalable Tree Boosting System, In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining ­ KDD ’16 The 22nd ACM SIGKDD International Conference, San Francisco, California, USA, ACM Press https://doi.org/10.1145/2939672.2939785 Chesney, E., Goodwin, G M., & Fazel, S (2014) Risks of all­cause and suicide mortality in mental disorders: A meta­review World Psychiatry, 13(2), 153–160 https://doi.org/10.1002/wps.20128 Chollet, F Et al (2015) Keras: The Python Deep Learning library https://keras.io Cook, N R (2007) Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction Circulation, 115(7), 928–935 https://doi.org/10.1161/CIRCULATIONAHA.106.672402 PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 24 Couronné, R., Probst, P., & Boulesteix, A.­L (2018) Random forest versus logistic regression: A large­scale benchmark experiment BMC Bioinformatics, 19(1) https://doi.org/10.1186/s12859­018­2264­5 Dietterich, T G., Ashenfelter, A., & Bulatov, Y (2004) Training conditional random fields via gradient tree boosting, In Twenty­first international conference on Machine learning ­ ICML ’04 Twenty­First International Conference, Banff, Alberta, Canada, ACM Press https://doi.org/10.1145/1015330.1015428 Earle, K A., Forquer, S L., Volo, A M., & McDonnell, P M (1994) Characteristics of Outpatient Suicides Psychiatric Services, 45(2), 123–126 https://doi.org/10.1176/ps.45.2.123 Fawcett, T (2006) An introduction to ROC analysis Pattern Recognition Letters, 27(8), 861–874 https://doi.org/10.1016/j.patrec.2005.10.010 Fazel, S., & Danesh, J (2002) Serious mental disorder in 23 000 prisoners: A systematic review of 62 surveys The Lancet, 359(9306), 545–550 https://doi.org/10.1016/S0140­6736(02)07740­1 Franklin, J C., Ribeiro, J D., Fox, K R., Bentley, K H., Kleiman, E M., Huang, X., Musacchio, K M., Jaroszewski, A C., Chang, B P., & Nock, M K (2017) Risk factors for suicidal thoughts and behaviors: A meta­analysis of 50 years of research Psychological Bulletin, 143(2), 187–232 https://doi.org/10.1037/bul0000084 Gunderson, J G., Weinberg, I., & Choi­Kain, L (2013) Borderline Personality Disorder FOCUS, 11(2), 129–145 https://doi.org/10.1176/appi.focus.11.2.129 Handley, T E., Hiles, S A., Inder, K J., Kay­Lambkin, F J., Kelly, B J., Lewin, T J., McEvoy, M., Peel, R., & Attia, J R (2014) Predictors of Suicidal Ideation in Older People: A Decision Tree Analysis The American Journal of Geriatric Psychiatry, 22(11), 1325–1335 https://doi.org/10.1016/j.jagp.2013.05.009 Hastie, T., Tibshirani, R., & Friedman, J H (2001) The elements of statistical learning: Data mining, inference, and prediction New York, Springer PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 25 Heisel, M J., Duberstein, P R., Lyness, J M., & Feldman, M D (2010) Screening for Suicide Ideation among Older Primary Care Patients The Journal of the American Board of Family Medicine, 23(2), 260–269 https://doi.org/10.3122/jabfm.2010.02.080163 Hong, V (2016) Borderline Personality Disorder in the Emergency Department: Good Psychiatric Management Harvard Review of Psychiatry, 24(5), 357–366 https://doi.org/10.1097/HRP.0000000000000112 Horton, M C (2018) Screening for the risk of self­harm in an adult offender population (PhD thesis) University of Leeds http://etheses.whiterose.ac.uk/id/eprint/20713 Horton, M C., Dyer, W., Tennant, A., & Wright, N M J (2018) Assessing the predictability of self­harm in a high­risk adult prisoner population: A prospective cohort study Health & Justice, 6(1) https://doi.org/10.1186/s40352­018­0076­3 Jaimes, F., Farbiarz, J., Alvarez, D., & Martínez, C (2005) Comparison between logistic regression and neural networks to predict death in patients with suspected sepsis in the emergency room, 9(2), https://doi.org/10.1186/cc3054 Jung, J S., Park, S J., Kim, E Y., Na, K.­S., Kim, Y J., & Kim, K G (2019) Prediction models for high risk of suicide in Korean adolescents using machine learning techniques (V De Luca, Ed.) PLOS ONE, 14(6), e0217639 https://doi.org/10.1371/journal.pone.0217639 Kessler, R C., Stein, M B., Petukhova, M V., Bliese, P., Bossarte, R M., Bromet, E J., Fullerton, C S., Gilman, S E., Ivany, C., Lewandowski­Romps, L., Millikan Bell, A., Naifeh, J A., Nock, M K., Reis, B Y., Rosellini, A J., Sampson, N A., Zaslavsky, A M., & Ursano, R J (2017) Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) Molecular Psychiatry, 22(4), 544–551 https://doi.org/10.1038/mp.2016.110 Khurana, U., Turaga, D., Samulowitz, H., & Parthasrathy, S (2016) Cognito: Automated Feature Engineering for Supervised Learning, In 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW) 2016 IEEE 16th International PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 26 Conference on Data Mining Workshops (ICDMW), Barcelona, Spain, IEEE https://doi.org/10.1109/ICDMW.2016.0190 Lundberg, S M., & Lee, S.­I (2017) A Unified Approach to Interpreting Model Predictions In I Guyon, U V Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, & R Garnett (Eds.), Advances in Neural Information Processing Systems 30 (pp 4765–4774) Curran Associates, Inc Mann, J J., Apter, A., Bertolote, J., Beautrais, A., Currier, D., Haas, A., Hegerl, U., Lonnqvist, J., Malone, K., Marusic, A., Mehlum, L., Patton, G., Phillips, M., Rutz, W., Rihmer, Z., Schmidtke, A., Shaffer, D., Silverman, M., Takahashi, Y., … Hendin, H (2005) Suicide Prevention Strategies: A Systematic Review JAMA, 294(16), 2064 https://doi.org/10.1001/jama.294.16.2064 Mann, J J., Ellis, S P., Waternaux, C M., Liu, X., Oquendo, M A., Malone, K., Brodsky, B S., Haas, G L., & Currier, D (2008) Classification trees distinguish suicide attempters in major psychiatric disorders: A model of clinical decision making The Journal of Clinical Psychiatry, 69(1)pmid 18312034, 23–31 McHugh, C M., Corderoy, A., Ryan, C J., Hickie, I B., & Large, M M (2019) Association between suicidal ideation and suicide: Meta­analyses of odds ratios, sensitivity, specificity and positive predictive value BJPsych Open, 5(2) https://doi.org/10.1192/bjo.2018.88 Naud, H., & Daigle, M S (2013) How to improve testing when trying to predict inmate suicidal behavior International Journal of Law and Psychiatry, 36(5­6), 390–398 https://doi.org/10.1016/j.ijlp.2013.06.010 Neumann, A., Holstein, J., Le Gall, J.­R., & Lepage, E (2004) Measuring performance in health care: Case­mix adjustment by boosted decision trees Artificial Intelligence in Medicine, 32(2), 97–113 https://doi.org/10.1016/j.artmed.2004.06.001 Paris, J (1997) Antisocial and borderline personality disorders: Two separate diagnoses or two aspects of the same psychopathology? Comprehensive Psychiatry, 38(4), 237–242 https://doi.org/10.1016/S0010­440X(97)90032­8 PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 27 Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, E (2011) Scikit­learn: Machine Learning in Python Journal of Machine Learning Research, 12, 2825–2830 Pestian, J P., Sorter, M., Connolly, B., Bretonnel Cohen, K., McCullumsmith, C., Gee, J T., Morency, L.­P., Scherer, S., Rohlfs, L., & the STM Research Group (2017) A Machine Learning Approach to Identifying the Thought Markers of Suicidal Subjects: A Prospective Multicenter Trial Suicide and Life­Threatening Behavior, 47(1), 112–121 https://doi.org/10.1111/sltb.12312 Podlogar, M C., Rogers, M L., Stanley, I H., Hom, M A., Chiurliza, B., & Joiner, T E (2018) Anxiety, depression, and the suicidal spectrum: A latent class analysis of overlapping and distinctive features Cognition and Emotion, 32(7), 1464–1477 https://doi.org/10.1080/02699931.2017.1303452 Quinlan, J (1987) Simplifying decision trees International Journal of Man­Machine Studies, 27(3), 221–234 https://doi.org/10.1016/S0020­7373(87)80053­6 Raczko, E., & Zagajewski, B (2017) Comparison of support vector machine, random forest and neural network classifiers for tree species classification on airborne hyperspectral APEX images European Journal of Remote Sensing, 50(1), 144–154 https://doi.org/10.1080/22797254.2017.1299557 Raeder, T., Forman, G., & Chawla, N V (2012) Learning from Imbalanced Data: Evaluation Matters (D E Holmes & L C Jain, Eds.) Data Mining: Foundations and Intelligent Paradigms, 23, 315–331 https://doi.org/10.1007/978­3­642­23166­7_12 Ribeiro, J D., Huang, X., Fox, K R., Walsh, C G., & Linthicum, K P (2019) Predicting Imminent Suicidal Thoughts and Nonfatal Attempts: The Role of Complexity Clinical Psychological Science, 7(5), 941–957 https://doi.org/10.1177/2167702619838464 Richards, J E., Whiteside, U., Ludman, E J., Pabiniak, C., Kirlin, B., Hidalgo, R., & Simon, G (2019) Understanding Why Patients May Not Report Suicidal Ideation at a Health Care Visit Prior to a Suicide Attempt: A Qualitative Study Psychiatric Services, 70(1), 40–45 https://doi.org/10.1176/appi.ps.201800342 PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING 28 Sacks, S., & Melnick, G (2011) Criminal Justice Drug Abuse Treatment Studies (CJ­DATS): The Criminal Justice Co­Occurring Disorder Screening Instrument (CJ­CODSI), 2002­2008 [United States]: Version Inter­university Consortium for Political and Social Research https://doi.org/10.3886/ICPSR27963.v1 Saito, T., & Rehmsmeier, M (2015) The Precision­Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets (G Brock, Ed.) PLOS ONE, 10(3), e0118432 https://doi.org/10.1371/journal.pone.0118432 Stoppe, G., Sandholzer, H., Huppertz, C., Duwe, H., & Staedt, J (1999) Family physicians and the risk of suicide in the depressed elderly Journal of Affective Disorders, 54(1­2), 193–198 https://doi.org/10.1016/S0165­0327(98)00149­9 Takeuchi, T., & Nakao, M (2013) The relationship between suicidal ideation and symptoms of depression in Japanese workers: A cross­sectional study BMJ Open, 3(11), e003643 https://doi.org/10.1136/bmjopen­2013­003643 Tan, S., Caruana, R., Hooker, G., Koch, P., & Gordo, A (2018) Learning Global Additive Explanations for Neural Nets Using Model Distillation arxiv 1801.08640 Retrieved May 17, 2019, from http://arxiv.org/abs/1801.08640 Walsh, C G., Ribeiro, J D., & Franklin, J C (2017) Predicting Risk of Suicide Attempts Over Time Through Machine Learning Clinical Psychological Science, 5(3), 457–469 https://doi.org/10.1177/2167702617691560 Wang, L., Ross, C A., Zhang, T., Dai, Y., Zhang, H., Tao, M., Qin, J., Chen, J., He, Y., Zhang, M., & Xiao, Z (2012) Frequency of Borderline Personality Disorder Among Psychiatric Outpatients in Shanghai Journal of Personality Disorders, 26(3), 393–401 https://doi.org/10.1521/pedi.2012.26.3.393 Were, K., Bui, D T., Dick, Ø B., & Singh, B R (2015) A comparative assessment of support vector regression, artificial neural networks, and random forests for predicting and mapping soil organic carbon stocks across an Afromontane landscape Ecological Indicators, 52, 394–403 https://doi.org/10.1016/j.ecolind.2014.12.028 World Health Organization (2014) Preventing suicide: A global imperative (S Saxena, E G Krug, & O Chestnov, Eds.) Geneva, World Health Organization ... prevention, borderline personality disorder, machine learning, prediction, classification, bpd, tree boosting PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING Predicting suicidal behaviour without asking. .. LEARNING Predicting suicidal behaviour without asking about suicidal ideation: Machine Learning and the role of Borderline Personality Disorder criteria Suicide is a major global health issue, with... in parentheses APD = antisocial personality disorder, diagnostic criteria ≥ 3; BPD = borderline personality disorder, diagnostic criteria ≥ PREDICTING SUICIDAL BEHAVIOUR USING MACHINE LEARNING