Evaluation of the Tech Schools Initiative
“Tech Schools help students think creatively, work together to solve problems and help prepare them for the future by delivering the advanced education and [science, technology, engineering and mathematics] STEM skills they need to compete in the future job market. They link secondary schools to industry to deliver innovative learning programs that challenge students to solve problems in a real-world context.” (Department of Education and Training Victoria, 2021).
Background
The Victorian Department of Education and Training (DET) is at the forefront of modernising STEM teaching. In Victoria, ten Tech Schools were established to complement the secondary curriculum by providing access to advanced technology, industry programs, and exposure to career pathways that secondary students may not otherwise access.
To demonstrate the value of the Tech Schools initiative, DET has partnered with Deakin University to design, implement, and verify a standardised evaluation system. It is guided by the Tech Schools Evaluation Framework, which describes short and medium-term outcomes, and six Key Evaluation Questions (KEQs).
During the first phase of the project in 2020, Deakin University developed an Evaluation Implementation Plan. The Evaluation Methodology is based on an Innovation Framework focusing on Content, Structure, and People. Deakin University developed a suite of tools, collected baseline, and case study data, qualitative and quantitative, and analysed data to measure outcomes.
Contribution by Intersect
Jerry, our eResearch Analyst at Deakin University, was engaged by the lead investigators to collaborate on the project and contribute primarily to the project’s data analytics and reporting components.
Data Analytics
A key part of Jerry’s contribution is providing support to Deakin researchers by analysing the data obtained in exit surveys of students, accompanying teachers, and teachers attending Professional Learning (PL). As part of this process, the Key Performance Indicators (KPI) and the corresponding evaluation questions from previous reports (2017 – 2019) were further refined and used in the evaluation process in 2020.
Reporting
Based on this framework, Jerry worked with the research team to develop a reporting system on how the Tech Schools perform against these KPIs, both at the individual Tech School and collective levels. Furthermore, secondary analyses were performed on the sub‐items (or facets) in the students and teachers feedback, focused on satisfaction, confidence, desire to re‐attend, student pathway indicators, and teacher intentions to change practice for each Tech School.
Outcomes
As a quantitative research expert, Jerry helped Deakin researchers to develop an easy-to-use report creation system using R and RMarkdown. This enables the stakeholders at DET and Deakin University to generate reports and updates for each Tech School by simply feeding the relevant data through the system.
The department is satisfied with the system, and Jerry continues to work with the research team to further improve and refine the design for subsequent data collections. In addition, Jerry also co-authored the Tech Schools Evaluation Annual Report (2020), which was used to facilitate strategic planning and decision-making by the department. The Year 2 Tech Schools Evaluation is currently underway.