Reporting Progress With Post-paper Digital Exams

By Andrew Fluck.

Computer-based exams have been used at the University of Tasmania since 2007 and in year 11/12 exams across the state since 2011. Both institutions have used the same USB-based eExam system. These developments have been picked up beyond Tasmania, and we are now working on the concept of Post Paper exams.

It may be interesting to know some of the background for the innovation. In educational contexts, students’ endeavours are rightly applauded by schools. However, workplaces recognise efficiency and achievement far more strongly. Therefore, our view has been that exams should be seen as a component of a balanced suite of assessments. Exams have the characteristic of providing every student with the same amount of time and resources to approach a problem. In addition, supervised exams assure the identity of each candidate, and this can be a crucial attribute for selecting from them to positions in industry, government or commerce. So, to make it quite clear, the work described in this article is about exams, but we see these to be just one component in assessments designed to be fit for purpose.

Mal Lee sees paper-based exams as inhibiting new directions in curriculum. This is probably very true, and the lack of digital assessment adoption by qualifications authorities and universities can be seen as one of the major problems for Australia going forward. The consequences of adopting changed assessment techniques could very well be a move from a population being groomed as digital consumers, to becoming individual creators and economic competitors of the future. Further arguments are being given about the role of computer-based exams towards the adoption of a curriculum transformed through the use of this technology (Hillier and Fluck 2013).

One of the logistic factors influencing the adoption of computer-based exams is the provision of equipment. When designing the eExam system, the author was quite clear that bring your own device (BYOD) was the only way to go. Institutions are unlikely to invest in equipment which only gets used a few times a year and would be needed in rather large numbers for periodic exams. However, if you are going to ask each candidate to bring their own computer, surely each and every one of these has a different suite of software or computing capability? This is a serious problem, especially if we seek to provide an equal context to every candidate to make some kind of comparative measurement of their achievements.

Therefore, the eExam system was designed to boot each candidate’s computer from a copy of the same USB stick. The USB stick contains a full operating system, tailored for the exam situation. Based on Ubuntu, it provides the crucial office tools of a word processor, spreadsheet, presentation application and graphics tools, etc. The USB can also be loaded with any particular software or PDF documents required for the set questions. Another reason for going down this individualised path is that individual equipment problems will not prevent an entire cohort of students from completing their assessment. With high stakes exams, this is a very important factor to be considered.

However, once we have a situation where the student has an entire operating system which is identical to that of all other candidates, we have a very interesting digital assessment context. As a small example, a project run a couple of years ago in primary schools provided every student with a copy of the MAPLE software. This is a very rich mathematical tool, and the students were able to become proficient with integral calculus using a computer for all the hard work. In this particular project, the students did a test on the computer based on a first year engineering university exam. They came out with good credit and distinction marks (Fluck, Ranmuthugala, Chin & Penesis, 2011).

It is important to distinguish the eExam system from some people’s initial assumptions of what computer-based exams might look like. Because experience is so limited, educators often conceive of computer-based exams as multiple choice questions. This is attractive, because the automatic marking of such questions has immediate positive benefits for teacher workload. However, a new way of looking at digital assessment should go far beyond this. Another view is sometimes a way of providing a word processor, instead of a pen. This also has implications for assessors. Instead of having to trawl through spider-like writing, the printed text on screen is much easier to read, and therefore mark. It is worth noting there is a degree of additional anonymity for each candidate, without that tell-tale handwriting. Examsoft and Exam4 are examples of this kind of facility, often flagged as the ‘armoured word processor’. They just reproduce paper-based exams in a new environment.

We need to go beyond this.

The pathway to a future digital assessment is important. Motivation, political backing, and business reputation are all part of the mix which will allow us to make the transition from paper to screen. If we are talking to schools in Tasmania about adopting the exam system, particularly independent schools, they stressed the importance of reputation. Their business model needs leavers to be demonstrably successful – especially in their academic years post-school. Thus, they train their students to succeed in the forms of assessment used at university. This can easily lead to a catch 22 situation where schools will not change unless the university changes.

In 2007, the eExam system was first used at the University of Tasmania. We used institutional laboratory computers, booting from CD ROMs. Since then, the eExam system has been developed both in Tasmania and at the University of Queensland to become much more sophisticated. We can prepare an exam on a USB stick which is duplicated using the digital equivalent of a photocopier. Students can boot their Macintosh or Windows PC from any of these USB copies. A key security feature is the display of the unique photograph (sometimes the examiner’s pet dog) on the desktop of every machine’s desktop. This provides non-technical supervisors an easy check to ensure every student has started with exactly the same digital resources. According to the exam context, we usually lock away access to any other digital storage device, such as the internal hard drive, so access to pre-prepared material is not possible. Also, we generally lock down any access to communication devices to make sure there is no collaboration between candidates, either through Bluetooth or via the internet.

Over the last seven years, the eExam system at the universities of Tasmania and Queensland has been used in history, mathematics, education, information technology, veterinary science, and law. Although this is only happening on a small scale (groups of up to 200, on three campuses and scattered around the world) it has provided a signal to the pre-tertiary sector.

In 2011, the Tasmanian Qualifications Authority used the eExam system to conduct a mid-year examination across the state for pre-tertiary Information Technology and Systems. Based on that experience, the end of year exams were also conducted state-wide using the eExam system. Because of the high stakes nature of this exam, it was important to have a strategy to fall back onto paper, therefore the questions were exactly the same as would have been set for a paper-based test. However, by 2012, the process was considered reliable enough to include some post paper questions and students were required to investigate a website (provided on the USB stick) as part of their problem solving. Moving on to 2013, the eExam system was given up and students required to answer questions with full access to the internet, but logging every interaction they made with outside parties as part of the assessment process. It is also worth noting Finland has made a decision to use a USB-based system for all their year 11/12 exams from 2020, with a pathway of adoption in different subject areas. One of the interesting things about the Finnish experience has been a competition to expose any vulnerabilities in the software before it becomes used more widely.

The whole point about digital assessment is that it provides new opportunities to work in new ways. Shown below is a snippet from a question that might conceivably be set in a post paper examination. Rather than being bound to a word processor, we can expect students to be grappling with powerful software which enhances their capabilities and requires them to think at higher levels. What is handed in on their USB stick may be an STL 3D design file, along with any word-processed rationale and photographic evidence.

A nationally funded project is currently developing a new version of the exam system, using Ubuntu 14.04 as its basis. This will give access to a very modern operating system which should be able to support newer hardware audio and video chips. It should also provide a pathway to use the BYOD equipment, even when it is running tamper-proof operating systems such as Windows 8, which makes it very difficult to use your personal equipment with any other kind of operating system except that from Microsoft.

As institutions and authorities begin to adopt new digital assessment techniques such as the eExam system, we look forward to a flowering of interest in the design of post paper assessments. These will not be the same for everyone, but will gradually help us to transform the curriculum so computers are seen as vital for learning as they are for the workplace.

Dr Andrew Fluck is a Senior Lecturer in Information Technology with the School of Education
Faculty of Education
at University of Tasmania.

References

The following two tabs change content below.

• Bio
• Latest Posts

Education Technology Solutions

Education Technology Solutions has been created to inspire and encourage the use of technology in education. Through its content, Education Technology Solutions seeks to showcase cutting edge products and practices with a view to expanding the boundaries and raising the standards of education curricula. It introduces teachers and IT staff to the latest products, services and developments in education technology with a view to providing practical how-to guidance designed to facilitate the integration of those products and services into the school environment in the most productive and beneficial manner possible.

Latest posts by Education Technology Solutions (see all)

• BenQ Launches First EDLA-Certified Interactive Displays for Education with Google Mobile Services (GMS) - November 17, 2023
• How AI technology is unlocking new opportunities for educators and pathways for learning - October 11, 2023
• A Strategic Implementation of Contemporary Digital Technologies - July 11, 2023