Evaluating Victorian Courts’ first
Self-determination Plan.
Assessing implementation, evaluating governance and measuring change and impact for First Peoples from Court Services Victoria’s Self-determination and Koori Employment Plans (2021-2025).
The brief.
Court Services Victoria is an independent statutory body that provides services and facilities to Victoria’s courts, the Victorian Civil and Administrative Tribunal (VCAT), the Judicial College of Victoria and the Judicial Commission of Victoria (collectively known as the Courts Group).
Court Services Victoria’s (CSV) Self-determination and Koori Employment Plans (2021-2025) outline the commitments of CSV and the wider Courts Group to improving cultural safety, embedding self-determination, First Peoples engagement and leadership in their operations and improving the attraction, recruitment and retention of First Peoples staff.
In preparation for the development of the next iteration of the Plans and to ensure accountability, CSV sought to understand the effectiveness of the Plans’:
implementation, in particular the extent to which actions have been completed, and as intended;
actions, in terms of achieving goals and intended outcomes;
administration, or the process of implementation, monitoring, reporting and oversight.
To ensure a robust and independent evaluation, Gyongyi was engaged to conduct a full service evaluation from the development of an appropriate and culturally safe evaluation approach to data collection, analysis and the development of final reports and artefacts.
The approach.
To ensure the evaluation centred First Peoples voices and embedded cultural safety and self-determination in every aspect, the approach was collaborative at its core, working with CSV’s Dhumba Murmuk Djerring Unit (DMDU) every step of the way to draw on deep and expert knowledge across the Courts Group, leverage work and data collection to date and design the most suitable approach to measurement and engagement.
Given the significant number of complementary and interdependent actions across both Plans, and the overlap of goals, intended outcomes and key stakeholders involved in implementation, the Plans were evaluated together, considering the three dimensions of effectiveness systematically. A separate evaluation approach was designed to consider each dimension of effectiveness, with data collection synergised to minimise the impact of consultation and engagement with key stakeholders. The design also specifically enabled the capture of a global view of the effectiveness of the Plans across these three dimensions, as well as a comparative view through individual Courts Group member perspectives to capture richer learnings for the next iteration of the Plans.
Effectiveness of implementation
Effectiveness of implementation was assessed based on a review of cumulative reporting throughout the lifespan of the Plans on the completion of actions based on a rubric-based approach. In addition to completion, the review also focused on evaluating whether individual actions were implemented as intended and becoming part of business as usual (BAU) and/or core business throughout the Courts Group, and/or still outstanding at the end of the Plans.
Key methodologies used included desktop reviews of existing reports, and supplementary consultation with relevant stakeholders.
Effectiveness of actions
Effectiveness of actions was assessed using an outcomes-based approach considering whether the consequences of the actions implemented contributed to achieving intended goals around cultural safety, self-determination, accountability and First Peoples employment.
To do so, a theory of change was developed to outline key assumptions around how actions were believed to contribute to, and achieve intended outcomes over time. Given common themes and the overlap of actions across both Plans and individual Courts Group member commitments, the evaluation focused not on the effectiveness of individual actions per se, but the effectiveness of key action categories - similar types of actions - as levers of change, as well as immediate, short and where possible, medium term outcomes in key priority areas. A focus on action categories enabled a streamlined approach for the consideration of all actions under the Plans, as well as a comparative view of outcomes across the Courts Group, including the ability to highlight different outcomes arising from different weight or focus placed on particular types of actions as part of the Plans.
Long term outcomes were out of scope given the timing of the evaluation prior to the completion of the Plans, however the cumulative impact of early actions could provide indicative insight into the effectiveness of actions on the medium term. As sourcing input from First Peoples accessing court services, stakeholders and the wider Community was out of scope of this evaluation, for short and medium term outcomes intended to benefit these audiences, the evaluation relied on existing datasets, if available, and drew on First Peoples staff perspectives to provide an indicative assessment (noting that the Plans were intended to deliver improved outcomes for these audiences on the long term).
Key methodologies used included desktop reviews of existing reports and datasets, surveys and consultations - individual interviews or group sessions - with a focus on testing the theory of change, establishing the most significant change and realist/contribution analysis of causation.
Effectiveness of administration
Effectiveness of administration was assessed based on reviewing development, roll out, monitoring, reporting and oversight processes, considering challenges encountered, their implications on the implementation of actions and process adjustments made throughout the lifecycle of the plan.
Key methodologies used included surveys, targeted consultations and process reviews.
Services provided.
Full service evaluation
Program logic / theory of change development
Evaluation framework development
Co-design, testing and refinement of rubric-based framework and development and refinement of criteria and scoring levels
Design and implementation of data collection tools - surveys (x2) and interviews, workshops, consultations and drop-in sessions (28 in total)
Review and analysis of documents and datasets (67 in total)
Synthesis and analysis of data, including collaborative sense-making sessions
Development of detailed findings report and supporting appendices, summary report and concise findings factsheets to support communication of findings
Development of dynamic progress trackers and dashboards to support monitoring for outstanding actions
Development of case studies
The outcomes achieved.
The evaluation provided a clear global and comparative view of progress towards the implementation of the Plans across the Courts Group, highlighting areas of significant progress above and beyond commitments, systemic change in progress as well as outstanding gaps requiring urgent continued and targeted focus going forward
Findings around administration also highlighted opportunities to incorporate recommendations from the Yoorook Justice Commission’s reports around governance and oversight, aligning to the wider policy context of Treaty in Victoria.
Moving forward, the recommendations alongside the theory of change developed provided a clear foundation to build upon for the development and effective monitoring of future Plans, articulating and validating key assumptions around the impact of particular actions and how they contribute to intended outcomes, and the circumstances in which they are most likely to be effective.
“I have had the pleasure of working with Gyongyi on several projects and she continually delivers high quality work, effective communication, and an incredible level of understanding of key pressures and issues to consider. Gyongyi strives to ensure that cultural protocols are adhered to and that First Nations voices are amplified and centred throughout our projects.”
— Koori Policy Lead, Court Services Victoria Dhumba Murmuk Djerring Unit
Want to learn more? Let’s chat about how we can help you.
No challenge is too big or small. Whether you’re looking for specific advice or a sounding board to explore potential options, we can help.