By: Shelley Kinash, Vishen Naidu and Kayleen Wood.
One of the most important ways of evaluating teachers and education is by asking the students. Near the end of each semester, university students (and an increasing number of primary and secondary students) are handed forms to evaluate their teachers and classes. Learners indicate extent of agreement with statements such as – my educator helps me understand difficult concepts. There is a comment box for students who wish to elaborate. Student evaluation of teaching is important because it provides an indication of whether teacher intentions are meeting the mark with students. Many constructive ideas for positive changes come from the learners themselves.
Most evaluation is obtained through students filling out paper forms. The fill-in-the-circle (Likert) items are scanned into a computer for tabulation. Comments are manually transcribed. There are numerous problems with this paper approach. Learners who are not present when the forms are handed out do not have an opportunity to evaluate the course or teacher. Students usually complete the forms in a hurry and do not have time for reflection and consideration.
A common student fear is that the teacher will recognise their handwriting and they’re therefore reluctant to offer any criticisms. There are documented cases of teachers timing the form distribution before difficult tests or immediately following activities that students are known to enjoy.
Scanning paper forms and transcriptions are also prone to handling error and are slow and cumbersome. Forms get stuck together when going through the scanner. Handwriting can be difficult to read, meaning that transcriptions are inaccurate. Evaluation reports often take so long to complete that teachers do not receive the feedback in a timely enough manner to make changes to the next course run.
Bond University therefore decided to take student teaching evaluation online (electronic teacher evaluation or eTEVAL for short). The advantages of eTEVAL are that students can provide their feedback from the convenience of their computers where and when they are ready. The process is standardised and objective. Data analysis and resulting reports are fast and efficient. Both descriptive statistics and qualitative analysis are used to interpret the student feedback. Closing-the-loop can be completed sooner and is based on the student experience.
A committee considered multiple packages before deciding on EvaluationKIT (www.evaluationkit.com). This system was chosen for three main reasons. First, it is a Blackboard Building Block, which means it has a single sign-on with the Learning Management System (LMS). Students receive an email with a link to their eTEVALs. The data to determine which students evaluate which courses is already present in the LMS. Second, EvaluationKIT achieved higher security ratings than any other of the considered systems. Third, the personnel of this particular company are responsive and willing to make modifications to the system to customise to Bond University’s particular context and needs.
Before rolling out eTEVALs with all students in all courses across the university, Bond University ran a pilot project with volunteer teachers and learners and sought feedback from diverse stakeholders, including the students. In other words, students provided evaluation on student evaluation of teaching. The results showed an overwhelming preference for electronic over paper-based teaching evaluation. The reasons included increased time and convenience, and the perception of enhanced anonymity. Students expressed a feeling of validation by the increased emphasis and changes to evaluation processes. They said that filling out the forms often feels futile, in that in the past, they seldom heard what happened to the feedback they provided. Students shared that they were encouraged by the implicit message in evaluating the evaluation. The message they heard was that student feedback matters.
Following the pilot project, a thorough review of the system (EvaluationKIT) was conducted, prior to to a full scale roll out. The Office of Quality, Teaching, and Learning worked closely with EvaluationKIT to implement five key customisations to enhance the eTEVAL system.
- The decision to make eTEVALs mandatory required a system functionality that encouraged participation, while acting as a sanction that prevented students from gaining full access to their iLearn content. To address this, the first customisation was a ‘pop-up’ notification, which prompts students that they have TEVALs (teacher evaluations) to complete. The two options on the pop-up are to complete TEVALs or “do it later.” The latter option temporarily disables the pop-up to allow students to quickly access content.
- The second customisation was the option to skip each TEVAL, but in doing so, the student will be prompted to tick a box that reads “I have considered completing the TEVAL for this subject and have chosen NOT to complete.” The student will also need to provide some rationale for their decision.
- The third customisation involved the development of a function to create groups so that separate TEVALs can be accurately identified in special circumstances such as having more than one teacher per course.
- A fully automated report builder was also integrated into the system, which allows for faculty and educator level reports to be generated instantly. The report builder allows the user to produce comprehensive reports using filters and drop-down menus. The reports can also be summarised with the use of text and graphical data presentations.
- The fifth customisation was the development and integration of a comprehensive qualitative data analysis package. Using Clarabridge, analysis of the comment field text is fully automated, and is then transferred back into EvaluationKIT for the results to be viewed within the report builder.
Teaching evaluation at Bond is a straightforward process. Students access their eTEVALs through entering their online course page or clicking on a link sent to their email. There are two separate evaluations. Both surveys use a five-point Likert scale including a ‘not applicable’ response. The teacher survey is administered for each class a student is enrolled in, and consists of ten questions with two open-text comment boxes. In instances where the same course is offered more than once a year, the course evaluation is run at least once, and is comprised of five questions and an open text comment box.
TEVALs are available for students to complete between Weeks 10 and 14, allowing students to submit their evaluations through the examination period. Keeping the evaluations open over this period allows for two things to occur: a fairer, more complete evaluation of both the course and teacher, which also takes into consideration all assessment items including end of year exams, and second, greater opportunities of achieving significant response rates. Teacher and Faculty level reports are accessible from the first week of the following semester, which ensures student grades are not influenced by the feedback provided, and that feedback is provided in time to respond with constructive changes.
Throughout the entire transition process from the paper based to the electronic model, it was essential that staff and students were kept up to date about the changes taking place. Posters, digital signage, social media and newsletter advertisements were run. An essential part of the communication plan was to involve the students, primarily through the Student Association. In addition, teachers were encouraged to actively communicate with their students about the significance of TEVALs and the eTEVAL process and to remind them from time to time in class to complete their outstanding TEVALs. Educators were provided with short PowerPoint presentations that could be used in class to create awareness and an overview of the new system.
Evaluation of teaching, electronic or not, is here to stay. The final section of this article addresses teachers’ reactions and the emotional (psychological) place they come from, why this is so and how to best manage the impact of evaluation.
The negative perception, culture and marketing of evaluation of teaching sees most teachers assume one of four default positions upon the release (dissemination) of the evaluation results. Described by Arthur (2009) as shame, blame, tame or reframe, each of these derives from a deficit model of professional evaluation. Each reaction assumes the worst and positions recipients as going into defensive mode. Shame or embarrassment manifests in self-doubt and loss of confidence. Blame is about externalising factors considered out of the teacher’s control. Tame is also an approach of putting the onus on someone else, as this reaction to evaluation is to decide that it is the students who need to change. Of the four, only the last, reframe, has any positive outcome possibility attached, and then only with a fix-it mentality.
But why has this negative culture grown up around teaching evaluation? Perhaps it is in large part due to the historic use of the results, including professional development reviews, promotion checklists and evidence of problems. These are all elements that resemble a big stick. Evaluations are often implemented as the biggest, most quantifiable stick available. They are touted as efficient, consistent, reliable, valid … impersonal?
Teaching evaluations by themselves only tell the story from one perspective. Schools and universities need to be transparent and active in including other stakeholders in the observation, as well as learner outcomes, the actual results of the students, leading to a BETTER evaluation (Smith, 2008). In a culture of positive change, student feedback is not the only source of informing teaching improvement. Formal processes are put into place to match peer-to-peer to give teachers opportunities to observe
one another in the classroom, have reflective conversations and share ideas for moving forward. Evaluation of teaching can be the beginning of something great. The process and result are the conversations that develop a scholarly approach to teaching and learning, or in other words, the gathering of evidence around teaching.
In summary, what is missing in most teaching evaluation systems is the starting point of fame, as the fifth (or rather the first) stance on evaluation, where a positive and appreciative perspective allows for the acknowledgement, reflection and celebration of teaching done well, as evidenced by learners and peers telling teachers so. From here, evaluations can form part of a bigger picture of assessing and improving teaching. They can be used for professional development, trying out new approaches (teaching methods and strategies), and a starting point for discussion and growth, not an end point or fate accompli. A new appreciative inquiry culture needs to be fostered by teaching and learning champions to begin the conversation with the elements of curriculum and pedagogy that went well, moving to points for consideration and then ideas for improvement, and deciding how we are going to support teachers to implement the ideas, including those offered by students.
References
Arthur, L. (2009). From performativity to professionalism: lecturers’ responses to student feedback. Teaching in Higher Education, 14(4), 441-454.
Kinash, S., Knight, D. & Hives, L. (2011). Student perspective on electronic evaluation of teaching. Studies in Learning, Evaluation, Innovation and Development, 8(1), 86-97. FOR 1303
Smith, C. (2008). Building effectiveness in teaching through targeted evaluation and response: connecting evaluation to teaching improvement in higher education. Assessment & Evaluation in Higher Education, 33(5), 517-533.
Kayleen Wood (kawood@bond.edu.au) is the Academic Developer in Quality, Teaching, and Learning at Bond University, Queensland, Australia. Kayleen’s qualifications are in business economics and human resource management, with her current study and research in higher education.
Vishen Naidu is a project coordinator in the Office of Quality, Teaching, and Learning at Bond University, Queensland. Vishen’s primary role is to oversee the integration of electronic teaching evaluations (eTEVALs) at Bond. His qualifications are in business, marketing and international business.
Shelley Kinash, PhD is the Director of Quality, Teaching and Learning at Bond University on the Gold Coast, Australia.
Shelley Kinash
Latest posts by Shelley Kinash (see all)
- Mum, I’m going to the Olympics: Educational Technology As An Enabler - October 23, 2020
- How to Design & Deliver Quality Online Education - June 1, 2019
- How to Achieve Industry Relevance and Student Engagement in Online Learning - March 1, 2019
You must be logged in to post a comment.
There are no comments
Add yours