In the same way, it’s important to be clear about what our evaluation criteria are, and how important each one is. If we can clarify this before we collect and analyse our data, we give ourselves a much better chance of being able to make a clear and sound judgement.

Broad criteria

Broad evaluation criteria help guide the direction of the evaluation. Imagine two colleagues reflecting on the new timetable arrangements for senior students.

Teacher 1: 'It’s great! Smooth as silk. Everyone knows where they need to be and it works really well with the rest of my classes.'

Teacher 2: 'I totally disagree – it’s a mess! The whole idea was to free up time for one-to-one feedback on their major works, but no one’s booking in times. My students are getting less feedback now than they were before.'

The reason why these colleagues are at odds in their evaluation of is not because they disagree about the facts, but because they are prioritising different evaluation criteria. Teacher 1’s evaluation (It’s great!) is based on the administrative efficiency of the changes. Teacher 2’s evaluation (It’s a mess!) is based on the impact on the amount and quality of feedback that students are receiving.

The list below outlines some of the main evaluation criteria that might be included in an evaluation.

• Effectiveness – The effectiveness of a program is the extent to which it achieves its intended objectives for the target audience. This might include changes in teaching practices, learning environments, student wellbeing, learning engagement or learning outcomes. Follow the link to Outcome evaluation for more information.
• Equity – Rarely does a program or project simply ‘work’ or ‘not work’ equally for everyone involved. In your assessment of program effectiveness, it’s important to be clear about which target audiences are a priority. The equity question is often about ensuring that disadvantaged groups profit at least equally to others. Equity can also have a process evaluation component, in terms of whether resources were distributed fairly and according to need in the implementation.
• Need, and responsiveness to need – A project or program might be deemed inappropriate if it is responding to an underlying need or problem that is a low priority, is rare or trivial (in that setting, or overall), is addressed by other initiatives, or has been misunderstood.
• Value for money – A project or program represents good return on investment when it achieves its objectives for a reasonable cost in terms of time, effort and resources. There are different ways of assessing return on investment, depending on our purpose and the availability of data. Follow the link below to Economic evaluation for more information.
• Efficiency – An efficient process is one that minimises wastage of time, effort or resources to get the job done. This is a more process-focused concept than return on investment. Follow the link below to Process evaluation and Economic evaluation for more information.
• Fidelity and consistency – 'Program fidelity' usually means the extent to which a structured program has been implemented as intended, according to the guidelines. However, the same terms can sometimes be used to mean equivalence from one region or site to another, for example, consistency of service delivery across NSW. Both of these may be relevant for a Process evaluation.
• Sustainability – The term 'sustainability' can mean different things, so it’s important to clarify in each context. The most common use of the term in evaluations is an operational one, where a sustainable program is one that can be maintained within the available resources. This is particularly the case when additional funding has been provided to introduce a program. The evaluation may examine whether the program and its benefits can be maintained beyond the initial funding period. Less often, people may use the term sustainability to talk about environmental impact or the longevity or persistence of the outcomes from a project or program over time (maintaining short term benefits into the future).

Specific criteria

Specific evaluation criteria help us measure the things that matter. Being specific about our evaluation criteria helps ensure that we aren’t working at cross purposes or collecting the wrong data for our evaluation.

For example, imagine the same two colleagues as above reflecting on the same new timetable arrangements for senior students.

Teacher 1: “It’s been great for student engagement. Absences are down, and my students almost always have the right books and materials for class.”

Teacher 2: “I’ve seen that too, and although it’s a positive change I still feel we have a way to go on the engagement front. I don’t think their attitudes to learning have changed much, and I’m not seeing any deeper engagement with the content.”

Again, the reason why these colleagues are at odds in their evaluation is not because they disagree about the facts. Rather, it’s because they mean different things by the term learning engagement.

Being specific about our evaluation criteria is particularly important by when it comes to talking about improved student outcomes, for example:

• If we are hoping to see an increase in student wellbeing, what aspects of student wellbeing are we hoping will change?
  • social wellbeing?
  • physical wellbeing?
  • mental health?
• If we say we are aiming for an increase in learning engagement, what aspects of learning engagement are we hoping will change?
  • Year 10-12 retention?
  • time on task?
  • attention to detail?
  • cognitive engagement with the content?
• If we say we are aiming for growth in learning outcomes, which key learning area are we seeking to strengthen? Or are we aiming for literacy and numeracy outcomes, which are cross key learning areas? Or are we seeking growth in skills like problem solving and higher order thinking, which are also cross-key learning areas?