Focus area 4: Evidence and Evaluation Evidence

In the optimal system, programs and services provide consistently high-quality supports for children and families. This focus area speaks of the importance of embedding a culture of evaluation in child and family mental health supports. This culture of evaluation includes:

1. Providing services and programs that have proven to be effective through robust evaluation
2. Ensuring that services are delivered and programs are implemented with high fidelity (i.e. they are implemented completely and with alignment to the way they were evaluated as effective)
3. Embedding ongoing feedback loops to check that programs are having the intended impacts in Australian contexts and to inform continued improvement. Importantly, these feedback loops must involve the children and families who use the services and participate in the programs.

The optimal system is comprised of evidence-based programs and services, where the impact on outcomes is monitored and evaluated, and funding is allocated accordingly. Key objectives in this focus area include improved collection and use of data, ensuring ongoing feedback and evaluation, and use of evidence to deliver quality services.

Objective 4.1 Meaningful data collection

Population data about children’s mental health and wellbeing are important for informing decision making in relation to policy and funding, as well as for measuring changes over time. We need reliable data to make sure that the resources are going where they are needed, and when changes are made, that they are having beneficial impacts for children and families. Meaningful data are also essential to monitoring whether systems and processes are changing in the right way, and whether the anticipated improvements are actually being achieved.

It is also important to collect data at three levels: process data, impact data and outcome data. Process data relates to implementation and includes indicators such as reach, engagement, satisfaction, and quality of programs and services. Impact data tracks ‘precursor’ variables that may indicate that things are heading in the right direction, such as changes in awareness, knowledge, attitudes/beliefs. Outcome data aims to measure whether changes occur in the fundamental issue, such as enhancements in mental wellbeing or the incidence and prevalence of mental health difficulties, disorders and suicidality, as well as non-health/mental health indicators such as interpersonal, academic and employment outcomes.

Children’s mental health is an area where key population data are missing. There is currently no regular national data collection or reporting regarding children’s overall mental health and wellbeing. To inform relevant policies and services, it would be helpful to have data relating to infancy (0-2 years old), preschool (3-5 years old) and primary school (6-12 years old). There are significant benefits to undertaking longer term research regarding children, including longitudinal studies, as outcomes may not be immediately evident. This is particularly true of younger age groups. Longitudinal data can examine how environmental experiences interact and affect outcomes, and highlight windows of development where early intervention would be most beneficial.

There has been increasing interest in perinatal mental health screening as a result of increased understanding about the prevalence of perinatal anxiety, stress and depression, particularly in mothers. While there is currently a Perinatal National Minimum Data Set, this does not include any data elements related to mental health (action 4.1.a). Additionally, Australia does not have a national approach to monitoring screening rates despite Australian clinical guidelines recommending universal antenatal screening of pregnant women.

Such monitoring would help inform strategies to increase perinatal screening (action 1.1.a). Where screening rates have been measured (based on use of the Edinburgh Postnatal Depression scale), screening rates are much higher in public than private settings, with a 71.3% overall screening rate in the six month period of data collection. In general, women were less likely to be screened if they were older (aged 36+), were wealthier, were single or separated or if they identified as Aboriginal or Torres Strait Islander.

During the preschool years, children typically have regular contact with health services such as GPs and child and maternal health nurses. During these contacts, services are able to check on a child’s progress against developmental milestones and facilitate routine vaccinations. These contacts may also provide opportunities to systematically gather information on the mental health and wellbeing of children (action 4.1.b). For the individual child, this would be beneficial in terms of connecting them to supports if needed. For the community more broadly, the information gathered during these contacts could better inform national approaches to policy and funding.

For school age children, primary schools provide a unique opportunity to gain insights into the mental health and wellbeing of our children. There is a well-known association between student wellbeing and educational outcomes, such as attendance, connectedness and engagement with school, and NAPLAN performance. However, there is a key gap in national data collection in relation to measures of student wellbeing in education settings. This information is important to both enable improved understanding of the development of students’ social and emotional skills and to inform how schools can best support their students’ wellbeing (action 4.1.c).

Difficulties accessing data across sectors such as education, justice and community health have been identified as a barrier to research in child mental health. In work undertaken by the Commission to develop a National Mental Health Research Strategy, this was identified as the top priority to better progress research in this domain. A collaborative and coordinated approach to data collection and sharing is required across the various sectors that play a role in children’s mental health, and there needs to be a mechanism to share data across different services with the consent of individuals. These barriers should be addressed to facilitate better quality research (action 4.1.d). In addition, ideally a standardised approach to best practice data collection would be taken, including standard data collection points across all states and territories. The availability of consistent measures would lead to more rigorous longitudinal studies by providing linkages across different stages of development across the life course. Data also needs to be accessible for the communities that are the subject of research. This includes translation into additional languages, consideration of needs of people with disability, and generally ensuring that participants are able to meaningful engage with data gathered from their community.

The service features that are important, even essential, to service users can differ between cultures and communities. This means that services designed in the context of one culture or community may not translate effectively into another.

This is evident when services designed from the perspective of non-Indigenous Australians are delivered to Aboriginal and Torres Strait Islander Australians. Aboriginal and Torres Strait Islander services report that often these services are rejected by communities as they fail to recognise the importance of factors such as culture, history, and community connection. For services used by Aboriginal and Torres Strait Islander children and families, the evaluation component needs to value Aboriginal approaches to evidence and expertise, and to be informed by Aboriginal and Torres Strait Islander ways of knowing, understanding and evaluating (actions 4.1.e and 4.1.f).

However, this represents a challenge to obtaining service funding, since there is often not a formal evidence base (for example, randomised controlled trials), despite services observing the benefits of their approaches within their communities. When collecting data on Aboriginal and Torres Strait Islander children, it is important to work with Aboriginal and Torres Strait Islander organisations.

For Aboriginal and Torres Strait Islander children, family and communities, the four key priority reform areas of the National Agreement on Closing the Gap 2020 are critical to improved social emotional wellbeing and mental health outcomes. These priorities are:

1. Formal partnerships and shared decision making
2. Building the community controlled sector
3. Transforming Government organisations
4. Shared access to data information at a regional level.

As noted in priority reform four from Closing the Gap, Aboriginal and Torres Strait Islander people should be supported to understand what data has been collected about them and how they can access this information. Additionally, Aboriginal and Torres Strait Islander organisations should be supported to collect and use their own data.

 

For Aboriginal and Torres Strait Islander communities, a lack of data sovereignty means that programs or policies fail to reflect Aboriginal and Torres Strait Islander priorities, values, cultures, worldviews and diversity. When data is limited or does not reflect the experiences of those who are the subject of the data, then interventions (such as services and programs) that are designed on the basis of this evidence are unlikely to be successful. This Strategy supports data capture and evaluation that is the result of a co-design process informed by Aboriginal and Torres Strait Islander perspectives and expertise (action 4.1.e) and development of measures that appropriately represent cultural conceptualisations of wellbeing in collaboration with the relevant communities (action 4.1.f). This includes better integration into data analysis of culture, cultural practices and experiences. In order to inform improvements, gap analysis, and track progress and outcomes for Aboriginal and Torres Strait Islander children and families.

Service needs for culturally and linguistically diverse, refugee and asylum seeker communities also differ. Many of these communities will have differing ideas and conceptions around mental health, family and the role of community. These, as well as the unique experiences and challenges of communities have direct impacts on what children and families need from services. Services should ensure that feedback is obtained on an ongoing basis from children and families who use them, to build the evidence base for effective service delivery and to facilitate the tailoring of services to the unique needs of communities (action 4.1.f).

Things we can build on

Longitudinal data are particularly useful for exploring developmental pathways and identifying risk and protective factors, as evidenced by the insights gained from the Longitudinal Study of Australian Children. This study explores a range of development and wellbeing questions and has provided insights in areas such as the impact of bullying on mental health, parent relationships with children and sleep. Commencing in 2003-04, data has been collected from two cohorts each year with study participants now in their teens and young adulthood.

Other Australian longitudinal studies of note include the Raine Study and the Mater-University of Queensland Study of Pregnancy (MUSP). The Raine Study was established in 1989 as one of the largest prospective cohorts of pregnancy, childhood, adolescence and adulthood in the world. Outcomes from this research have delivered findings on the link between sleep patterns in childhood and adult mental health, and maternal smoking on children’s physical and mental health. The MUSP began in 1981 as a prospective study of 8,556 pregnant women, initiated after their first clinic visit. Data collection has occurred across various time points for both mothers and children, with a new phase examining outcomes in the third generation (grandchildren of original participants) about to commence.

The NSW Child Development Study is a more recently commenced longitudinal study of child mental health and wellbeing in a cohort of children who were assessed using the Australian Early Development Census in 2009 and/or the Middle Childhood Survey in 2015. The study aims to obtain data about the development of the cohort during early and middle childhood in order to map patterns of resilience and vulnerability for later mental health, education, work, and other outcomes.

The Australian Early Development Census (AEDC) is a current nationwide data collection of early childhood development held every 3 years. Data are collected from teachers at the time children commence their first year of fulltime school. The AEDC has five domains: physical health and wellbeing; social competence; emotional maturity; language and cognitive skills (school-based); and communication skills and general knowledge.

The South Australian Wellbeing and Engagement Collection is a survey that collects information from students in years 4 to 12 about non-academic factors relevant to learning and participation, including emotional wellbeing, engagement with school, learning readiness and health and wellbeing out of school. The collection was initiated in 2013, and is ongoing.

The Australian Institute of Health and Welfare has recently included three perinatal mental health screening data items in their National Best Endeavours Data Set to be collected from 1 July 2020. This is the first step in getting the items into the Perinatal Minimum Dataset and jurisdictions are already working towards this goal.

As part of the Be You evaluation, the Commonwealth Department of Health is funding the University of Queensland to develop a costed plan for future longitudinal research study on national promotion, prevention, early intervention and prevention mental health programs.

Actions 4.1

    a. Expand the Perinatal National Minimum Data Set to include indicators of mental health screening in the weeks before and immediately after birth (see also action 1.1.a).

    b. Establish a national system for pooling key information related to child mental health (wellbeing indicators) gathered via routine developmental checks or vaccinations conducted during the preschool years.

    c. Establish a national minimum dataset on student wellbeing with annual collection conducted via schools and regular reporting on how schools are addressing any issues identified.

    d. Task the Inter-Departmental Committees proposed by this Strategy (see section 4.1) with resolving current barriers to relevant data sharing across sectors such as education, justice and community health, for the purposes of informing child mental health and wellbeing.

    e. Ensure that data capture and evaluation practices related to Aboriginal or Torres Strait Islander child mental health and wellbeing are consistent with the principles of co-design (i.e. with significant involvement from the communities represented in the data).

    f. Develop measures that appropriately represent cultural conceptualisations of wellbeing in collaboration with the relevant communities, where such measures do not currently exist.

Objective 4.3 High-quality research

The development of new approaches to meeting the mental health needs of young people has led to national reforms and the development of a framework for youth mental health research in Australia. However, coordinated multidisciplinary efforts involving community consultation have yet to be similarly applied to children under 12 years of age and there is an overall lack of trials in child mental health. The 0-12 years age range offers a unique window for research exploring mechanisms that may cause and maintain mental illness. This includes causal processes for specific diagnoses, and those processes that span across multiple disorders. In addition, transdiagnostic approaches offer an opportunity to guide clinicians when children appear with complex presentations.

Moreover, apart from general factors like severity and parental mental health, knowledge is lacking around risk factors for persistence of symptoms from childhood into adulthood. Consequently, there is a lack of treatment programs specifically targeting children at high-risk, who will likely need a more intense treatment program. There are also no large scale harmonised data collections in child development, or in child and adolescent mental health programs (apart from the routine outcome measures) to improve the identification of children at high-risk.

There are evidence-based interventions available for some childhood onset disorders, for example Cognitive Behavioural Therapy for anxiety disorders and pharmacological treatment for ADHD but there are also huge gaps in our knowledge. To illustrate, psychotherapy for depression seems less effective in children under 13 years, the choice of pharmacotherapy for children with ADHD is now a case of trial and error, and there are not many evidence based interventions for ASD after the early years (action 4.3.a). What works for individual children can also vary, and not every child will find the standard treatment modality acceptable or useful. In some cases, it may be appropriate to embed research capacity and capability into service provision.

Ongoing clinical trials can play an important role in improving best practice in treatment guidance and guidelines. It is argued that compared to other areas of medicine (for example, oncology), the resources and culture in relation to children who are unwell taking part in trials are lacking in mental health. Research is also beneficial for those who participate, beyond just global knowledge generation. It is known for example, that participants in clinical trials tend to have better outcomes than those with similar characteristics who just receive standard care. Yet, despite the paucity of knowledge about the best treatments for many of the most common mental illnesses, there is not a strong culture of research as part of standard clinical practice in child psychology, allied health and psychiatry. We should ensure that research into the mental health needs and treatment outcomes for children receives adequate priority for funding (action 4.3.b), and that specific research targets are the result of consensus driven priorities, such as those identified under the National Mental Health Research Strategy.

Current ethics processes often make research with children challenging and this should be addressed (action 4.3.c). There is a focus by Research Ethics Committees on children’s status as ‘vulnerable’, with frequently raised concerns about issues of consent, methods of recruitment, care and protection of participants, scientific design and confidentiality. While it is reasonable for ethics committees to be concerned about these issues, including children and families (or consumers generally) in the design and development of research questions and research methods would be a helpful strategy to help navigate these concerns (action 4.3.d). Doing so would also increase the likelihood that the subsequent research will address real world treatment issues and questions.

In the optimal system, children would receive measurement-based care where treatment is optimised based on ongoing feedback. Clinical measures of outcomes of treatment would be augmented by the collection of information on other factors, including family characteristics, school functioning, and biomarkers, to support analysis to identify which children do not fully recover and why. These results could also be used to set up treatment trials to target those children who are still struggling in order to improve outcomes.

Things we can build on

The Million Minds Mission (under the Medical Research Future Fund) has prioritised participatory mental health research into prevention and wellness, early intervention and treatment strategies for children, youth and emerging adults. To date, two major grants have been awarded for projects focused on child mental health, including Emergency Department management of acute mental health crises and translating evidence-based interventions into population-level digital models. Another three grants were awarded for projects focused on mental health in Aboriginal and Torres Strait Islander communities including; generation of Indigenous patient-centred, clinically and culturally capable models of mental health care and culturally appropriate models for improving mental health and wellbeing in Aboriginal and Torres Strait Islander young people.

The Ethics Research Involving Children (ERIC) project aims to encourage ongoing international collaboration around improving ethical research involving children. It developed in response to an identified need for ethical guidance to support researchers throughout their research processes, and to create common resources for research funders and ethics review committees. The ERIC resources are freely available online, including a monitored forum to promote questions, discussion and debate, along with a resource library and case studies from researchers examining actual ethical challenges across different contexts and research methodologies, and the response to them at the time they arose.

James Lind Alliance Priority Setting Partnerships (PSP) aim to identify and raise awareness of research questions directly relevant to consumers, carers and health practitioners. The partnership model uses a participatory process involving health and social care professionals, carers and families and those with lived experience of the health area in question. In 2018, they conducted a PSP focused on mental health in children and young people that helped to identify the top 10 unanswered research questions.

Actions 4.3

    a. Advocate for research into child mental health needs and treatment as a priority, arguing for child mental health research to receive funding that is comparable to child physical health.

     b. Identify research priorities for child mental health, incorporating a focus on priority populations and the current gaps in our collective knowledge regarding treatments. Allocation of funding must be aligned with these identified priorities.

     c. Review current research ethics approval processes, looking for ways to facilitate innovative research without compromising the safety of research participants.

     d. Require researchers to consult with children, family and relevant community members as part of the research design process, with the expectation that those involved are also informed of any results of the research.

The below table outlines some proposed indicators of change. If the objectives of this focus area are realised, we would expect to see improvements in the below.

Focus Area 4: Evidence and Evaluation – indicators of change

    4.1. Collection and use of data relating to child mental health and wellbeing

    4.2. Use of evaluation and sharing of outcomes relating to child mental health and wellbeing