Network replicability: a cross-cultural PTSD study across of clinical datasets

Let’s start with a teaser: This blog post is a summary of the first psychological network study ever looking at the replicability of network models in 4 datasets, with a total N of 2,782 (Preprint). The paper is fully reproducible, and includes the covariance matrices of the four datasets so you can start re-analyzing the data with network or factor models.

This is how the four networks look like:

I will discuss below (1) why research on replicability and generalizability is important, (2) why we should start focusing on estimating network and factor models in clinical datasets instead of community samples, and (3) discuss the paper in more detail.

Stability and replicability of network models

We’ve worked hard in the last year to tackle the challenge of stability or accuracy of network models. As explained in a recent tutorial blog post, the R-package bootnet [1] can be used to investigate how stable state-of-the-art network models such as the Ising Model or the Gaussian Graphical Model are. The package was updated recently and now can also be used for relative importance networks, and we are working hard to be able to incorporate time-series models in the future.

Now that we can look into the stability of network models — and can show that network parameters can be estimated accurately in large samples — the next question is whether data-driven and exploratory network models generalize and replicate across different samples. Interestingly, there is no single published paper that I am aware of that has done so specifically. Note that a few papers exist that compared subsamples of the same dataset, such as groups of individuals with different substance abuse disorders [2].*

So about a year ago, I set out to write a paper on network replicability, and found a dozen of collaborators in Denmark, the Netherlands, and Italy who were willing to help with this endeavour.

In addition to replicability, there is a second main limitation of the network literature so far: similar to the SEM literature, the majority of prior studies were carried out in community or subclinical samples, rendering network structures in clinical samples largely unknown. This is most pronounced for the PTSD literature where more than 10 network papers have been published, but no single paper in a purely clinical dataset. Which is why we investigated the replicability of PTSD networks in 4 clinical samples.

PTSD network replicability

I uploaded the preprint of the paper yesterday. The repository not only includes the paper itself, but also the covariance matrix of the four datasets, along with all network parameter estimates, symptoms means and standard deviations, and many other R objects to make the paper fully reproducible. While we cannot share the original datasets, the covariance matrices are sufficient to estimate network and factor models, and we encourage re-analysis of the data.

What did we do, and what did we find? We estimated state-of-the-art Gaussian Graphical Models in four clinical datasets of traumatized patients seeking treatment. For the first time, to our knowledge, we estimated four networks jointly, using the Fused Graphical Lasso (FGL) [3] implemented by Giulio Costantini who collaborated on the paper. The FGL improves network estimates by exploiting similarities among different groups in case such similarities emerge; otherwise, networks are estimated independently. We then compared the resulting networks in terms of network structure and centrality indices. Here is a short summary:

The network approach to psychopathology understands mental illness like Posttraumatic Stress Disorder (PTSD) as networks of causally interacting symptoms. The prior literature is limited in three aspects: studies estimated networks in one sample each, leaving open the question whether networks replicate across samples; studies estimated networks in primarily small samples that may lack power for reliable estimation; and studies examined community or subclinical samples, rendering the network structure in clinical samples unknown. In this cross-cultural multisite study, we estimated state-of-the-art regularized partial correlation networks of 16 PTSD symptoms across four datasets of traumatized patients (total N=2,782). Considerable similarities emerged, with high correlations between network structures (0.62 to 0.74) and centrality estimates (0.63 to 0.75). Only 1.7% to 6.7% of the 120 edges differed across networks. Despite sample differences, networks showed substantial similarities, suggesting that PTSD symptoms may be associated in similar ways. We discuss implications for generalizability and replicability.

Detailed results and figures

First, it was interesting to see that symptom profiles were fairly similar across datasets, despite the very different composition and background of the four samples (ranging from a sample of predominantly male military veterans to another dataset of severely traumatized refugees with up to 30% psychotic symptoms). The plot below also contains standard deviations because there were some ceiling effects in dataset four, with very severe symptomatology and lower standard deviations, with a negative correlation of mean and standard deviation which can be nicely seen in the plot (details in the paper).

Second, network structures were not exactly the same: the omnibus network comparison test (with the null-hypothesis that all 120 edges per comparison are exactly identical) was significant for all pairs of networks. Overall, however, considerable similarities emerged, and correlations between network structures and centrality indices were high (see paper for more details).

Third, we estimated (a) a cross-sample network and (b) a cross-sample variability network. These depict similarities and differences across the four datasets.

Conclusion and future directions

Overall, we were surprised by the similarities of both symptom profiles and network structures across the four datasets, given the considerable differences in sample compositions. We conclude the paper with a call for more replicability work in psychopathology factor and network models:

We therefore conclude that investing time in more thoroughly conducted cross-sample studies for both network and factor models is warranted in order to facilitate insights about replicability and generalizability. We hope the present paper will encourage more researchers to do so.

Citation:

» Fried, E. I., Eidhof, M. B., Palic, S., Costantini, G., Huisman-van Dijk, H. M., Bockting, C. L. H., Engelhard, I., Armour, C., Nielsen, A. B. S., Karstoft, K. (submitted). Replicability and generalizability of PTSD networks: A cross-cultural multisite study of PTSD symptoms in four trauma patient samples. DOI: 10.17605/OSF.IO/2T7QP. (Preprint and Supplementary Materials)

Full abstract:

Introduction. The network approach to psychopathology understands disorders like Posttraumatic Stress Disorder (PTSD) as networks of mutually interacting symptoms. The prior literature is limited in three aspects. First, studies have estimated networks in one sample only, leaving open the crucial question of replicability and generalizability across populations. Second, many prior studies estimated networks in small samples that may not be sufficiently powered for reliable estimation. Third, prior PTSD network papers examined community or subclinical samples, rendering the PTSD network structure in clinical samples unknown. In this cross-cultural multisite study, we estimate and compare networks of PTSD symptoms in four heterogeneous populations of trauma patients with different trauma-types, including civilian-, refugee-, combat-, post-war off-spring-, and professional duty-related trauma.
Methods. We jointly estimated state-of-the-art regularized partial correlation networks across four datasets (total N=2,782), and compared the resulting networks on various metrics such as network structure, centrality, and predictability.
Results. Networks were not exactly identical, but considerable similarities among the four networks emerged, with moderate to high correlations between network structures (0.62 to 0.74) and centrality estimates (0.63 to 0.75); only few edges differed significantly across networks.
Conclusion. Despite differences in culture, trauma-type and severity of the four samples, the networks showed substantial similarities, suggesting that PTSD symptoms may be associated in similar ways. We discuss implications for generalizability and replicability. A step-by-step tutorial is available in the supplementary materials, including all analytic syntax and all relevant data to make the paper fully reproducible.

References:

[1] Epskamp, S., Borsboom, D., & Fried, E. I. (2017). Estimating Psychological Networks and their Accuracy: A Tutorial Paper. Behavior Research Methods, 1–34. http://doi.org/10.3758/s13428-017-0862-1
[2] Rhemtulla, M., Fried, E. I., Aggen, S. H., Tuerlinckx, F., Kendler, K. S., & Borsboom, D. (2016). Network analysis of substance abuse and dependence symptoms. Drug and Alcohol Dependence, 161, 230–237. http://doi.org/10.1016/j.drugalcdep.2016.02.005
[3] Guo, J., Levina, E., Michailidis, G., & Zhu, J. (2011). Joint estimation of multiple graphical models. Biometrika, 98(1), 1–15. http://doi.org/10.1093/biomet/asq060

Footnote:

* There is also a recent preprint by Forbes et al. to be published in the Journal of Abnormal Psychology that aimed to tackle replicability, but it makes more sense to discuss the paper once the commentaries on the paper are published that — to my knowledge — will point to substantial problems in the analyses.