This is the second in a series of articles examining why online formative tests are X-Rays and may be of little use.
“Flawed data produce flawed conclusions, and thus flawed understanding of a student’s knowledge and progress.”
Multiple choice questions are used extensively in formative online tests as marking can be automated and feedback can be instant; excellent reasons for their use. True/False questions are also used by some. My personal opinion, and that of some others, is that they should be used rarely as the student effectively has a 50% chance of guessing the correct answer.
Limited time and a tight schedule
Well written multiple-choice questions can be very effective. However, good questions take a lot of time to write and require an experienced and knowledgeable author. Sometimes individuals and organisations have time constraints that negatively impact the ability to write quality of questions – a trade-off that may be considered acceptable. Consequently, formative tests can fall short of the mark.
Gaming the system
Students have also worked out that questions can have flaws and know there are ways to ‘game’ the system; to guess the correct answer without knowing the material. Lists of ‘rules’ are readily available online.
This is a list of ‘rules’ published in 2015. It took less than two minutes to find this list using an internet search.
There are even inexpensive online courses that state they are “The Complete Course on Guessing Multiple Choice Exams and How to Survive an Exam You Haven’t Prepared For!”
We must ensure that our questions can fend off these types of ‘hacks’.
Here is a list of some ‘rules’ developed after a few internet searches. I have tested these with some online quizzes and sometimes achieved 50% (and often much more). How well would your formative quizzes handle these?
Some ‘rules’ to game formative online quizzes
- Choose the longest answer.
- Choose the most specific (detailed/complex) answer.
- Choose an answer if it is the only one with a ‘double option’; a statement AND another statement; a statement BUT another statement. This is often the most specific answer.
- Choose ‘all of the above’ or ‘none of the above’ if they are available.
- If two answers express opposites, choose one or the other and ignore the other options.
- Never pick an answer that uses ‘always’ or ‘never’.
- If in doubt, choose C.
- Where more than one answer must be chosen, choose all except the ‘obviously incorrect’ one.
- If a True/False question, choose True.
Are there other possibilities to make formative quizzes more robust?
There are. Using some features of your online learning environment can help create more robust quizzes.
- Randomise the responses in each multiple-choice question. This means that Response A will not always be Response A. It may be B, C, D, etc., even for two students taking the test at the same time.
- Randomise the questions. Question 1 may be Question 7, or 3, or whatever for another student, or if the quiz is repeated.
- Use a question bank. Write a number of questions testing the same information and have the system randomly choose which question will be given to a student. This requires much more time to develop a larger number of effective questions, but the test will be much more effective.
These possibilities can result in two students who could be sitting beside each other, taking a quiz testing the same knowledge, and yet they would effectively be taking ‘different’ quizzes. Each quiz could have different questions covering the same concepts, in a different order, and with multiple choice responses randomised. This is obviously much better than having a student who repeats a quiz being presented with the same questions with the same responses in the same order as the original quiz.
Delay
Consider placing a delay when a student retakes a quiz.
I recently took an online quiz of ten questions and was able to retake the quiz immediately. The questions were in the same order as my first attempt but had the multiple-choice responses randomised/ in a different order. It was relatively easy to remember my previous choices and thus achieve a score that was not a good representation of my knowledge.
Is the time and effort to develop good online test questions and strategies worth it?
I believe it is. ‘Good’ questions in online formative (and summative) assessments
- allow students to better gauge their level of understanding.
- allow teachers to better gauge the knowledge of each student.
- provide insights for the organisation about student progress, providing better diagnostic information and enabling more effective intervention and remediation.
- Ensure student data is of value, rather than being misleading.
Nothing new, yet still vital
None of the points raised in this article are new. Guidelines on writing effective assessment questions are as old as education. The features mentioned on randomising responses and questions, using question banks and introducing delays on repeated quizzes are built into any modern LMS. Yet, it is amazing how these concepts need to be reinforced even in 2021.
It is now even more important that we create robust and effective online quizzes with the increase in online learning in the era of Covid 19.
Latest posts by Peter West (see all)
- The 20:20:60 Rule of eLearning Evolution - October 11, 2022
- Is your school ready for an LMS? – The Data Design Continuum (DDC) - August 25, 2022
- Is your school ready for an LMS? – The Course Design Continuum (CDC) - September 7, 2021
You must be logged in to post a comment.
There are no comments
Add yours