Science is unpredictable. In studies of psychotherapy, many factors can influence your study and your findings: the resources available to people in the study, the situation in which they are living, the involvement of family and friends, the therapists they are seeing, attitudes toward psychotherapy, and many more. Therefore, psychotherapies are contextual, meaning that they may or may not work the same way when these factors change. Of course, these factors do change across time, locations, and individuals. Take for an example a common psychotherapy strategy: cognitive restructuring or thinking about things more realistically (and often more positively) to help reduce distress. A child living in a safe home with loving parents and friends who thinks “my life is really hard” may find this strategy much more relevant than a child living in poverty or without strong social support who has the same thought. We cannot always expect findings of psychotherapy studies to replicate across time, individuals, or contexts.
At Shamiri Institute, we’ve always adopted a data-driven scientific approach precisely because science can be unpredictable. When we first designed the Shamiri intervention in 2018, we wanted to test if it works. The Shamiri intervention consists of three components, growth, gratitude, and values. We found that the intervention working in reducing youth depression and anxiety symptoms in a pilot randomized controlled trial (RCT) in 20181 as well as in a replication RCT in 2019.2
Building on this evidence, we asked ourselves if the effects were being driven by combining the three components together by each of the components. That is why we conducted a RCT in 2021 which compared the effects on depression, anxiety, and wellbeing of the Shamiri Intervention, consisting of growth, gratitude, and values components, to each of its components delivered alone (i.e. growth-only, gratitude -only, and values -only).1,3,42 Finally, these interventions were also compared to our usual Study-Skills control group, which only teaches youth study skills such as notetaking, and nothing thought to have a direct impact on symptoms of anxiety or depression.
In this trial, we found something unexpected: all the interventions led to improvements but there were no differences in depression, anxiety, and wellbeing among the five groups. In previous trials, we had always found that the Shamiri Intervention outperformed the Study-Skills control group significantly,1,2,5 and we had also found that growth and values alone significantly outperformed a Study-Skills control (though gratitude did not).6
This trial may have had unusual results because of an unusual context; it was conducted in early 2021 right when schools re-opened after a 9-month COVID lockdown in Nairobi. At this time, schools were expected to complete three years of work in two years, which dramatically increased students’ academic stress. Additionally, schools were under stress as they faced a local epidemic of fire-setting by students attempting to avoid exams. Therefore, we tested several factors that could have influenced findings, including study drop-out, fidelity to the psychotherapy protocol (how closely peer therapists followed the intervention protocols), quality of peer therapists, presence of fire-setting, and number of sessions attended; none of these additional factors changed the overall results.
Finally, we wondered: Was there suddenly no difference in this study between the Shamiri Intervention and the control because the Shamiri Intervention was less effective than before? Or because Study-Skills was more effective? The answer was clear: Study-Skills was much more effective than before! The Shamiri Intervention, on the other hand, was only slightly more effective than before.
I think the most likely explanation for this unexpected finding is that the context of very high post-COVID academic stress during this recent study made learning Study-Skills especially relevant to helping students feel better. Likely, students were especially stressed about academic performance, leading to increased symptoms of anxiety and depression like worries and sleep problems. Therefore, supporting students academically helped more than usual to reduce these symptoms. We cannot know this for sure without additional research, but the data supports this reasonable hypothesis.
The findings of this study could be seen as disappointing: of course, all scientists want their ideas to work out and their work to replicate! However, by digging into these unexpected findings and investigating reasons why they might have occurred, we learned something important: interventions focused on contextually relevant skills, like study skills in a time of academic stress, may sometimes rival psychotherapies at alleviating anxiety and depression.
This possibility has led us to undertake a novel study on contextually relevant life-skills interventions, focused on academic, social, and financial skills. Each participant in this study is assigned to one of these three life-skills interventions based on which is relevant to their current stressors. We will soon see whether this life-skills intervention works to treat anxiety and depression symptoms. Without our unexpected results, we would have taken much longer to pursue this new and promising idea of testing life skills interventions to help with symptoms of mental illness.
Overall, this story speaks to values including honesty, service to those with mental illness, truth-seeking, and curiosity, all of which are dear to me and to Shamiri Institute. My personal aim, and our aim at Shamiri, is to do useful work that benefits youth in need of mental health support. Therefore, it is key to be honest about results, even when they are not what we have found before or what we expect, since they take us closer to truths and to new ideas that may help these youth to thrive.
Note: This blog post is a reflection on a recently published journal article supported by the Templeton World Charity Foundation.
The full article citation is: Venturo-Conerly, K. E., Osborn, T. L., Rusch, T., Ochuku, B. K., Johnson, N. E., van der Markt, A., ... & Weisz, J. R. (2024). Testing the Shamiri Intervention and Its Components with Kenyan Adolescents During the COVID-19 Pandemic: Outcomes of a Universal, Five-Arm Randomized Controlled Trial. Journal of the American Academy of Child & Adolescent Psychiatry. https://doi.org/10.1016/j.jaac.2024.04.015
References:
1 Osborn, T. L., Wasil, A. R., Venturo-Conerly, K. E., Schleider, J. L., & Weisz, J. R. (2020). Group intervention for adolescent anxiety and depression: outcomes of a randomized trial with adolescents in Kenya. Behavior therapy, 51(4), 601-615. https://doi.org/10.1016/j.beth.2019.09.005
2 Osborn, T. L., Venturo-Conerly, K. E. , Arango, S., Roe, E., Rodriguez, M., Alemu, R., Weisz, J. R. (2021). Effect of Shamiri layperson-provided intervention vs study skills control intervention for depression and anxiety symptoms in adolescents in Kenya: a randomized clinical trial. JAMA Psychiatry, 78(8), 829-837. https://doi.org/10.1001/jamapsychiatry.2021.1129
3 Venturo-Conerly, K. E., Osborn, T. L., Rusch, T., Ochuku, B. K., Johnson, N. E., van der Markt, A., ... & Weisz, J. R. (2024, online). Testing the Shamiri Intervention and Its Components with Kenyan Adolescents During the COVID-19 Pandemic: Outcomes of a Universal, Five-Arm Randomized Controlled Trial. Journal of the American Academy of Child & Adolescent Psychiatry. https://doi.org/10.1016/j.jaac.2024.04.015
4 Venturo-Conerly, K. E., Osborn, T. L., Wasil, A. R., Le, H., Corrigan, E., Wasanga, C., & Weisz, J. R. (2021). Testing the effects of the Shamiri Intervention and its components on anxiety, depression, wellbeing, and academic functioning in Kenyan adolescents: study protocol for a five-arm randomized controlled trial. Trials, 22, 1-20. https://doi.org/10.1186/s13063-021-05736-1
5 Osborn, T. L., Rodriguez, M., Wasil, A. R., Venturo-Conerly, K. E., Gan, J., Alemu, R. G., ... & Weisz, J. R. (2020). Single-session digital intervention for adolescent depression, anxiety, and well-being: Outcomes of a randomized controlled trial with Kenyan adolescents. Journal of consulting and clinical psychology, 88(7), 657. https://doi.org/10.1037/ccp0000505
6 Venturo-Conerly, K. E., Osborn, T. L., Alemu, R., Roe, E., Rodriguez, M., Gan, J., ... & Weisz, J. R. (2022). Single-session interventions for adolescent anxiety and depression symptoms in Kenya: A cluster-randomized controlled trial. Behaviour research and therapy, 151, 104040. https://doi.org/10.1016/j.brat.2022.104040
