Presentation Using AI Tools to Create and Analyse Departmental Placement Tests more

Many universities outsource placement to “off-the-shelf” tests, but AI now lets schools produce localized, tailor-made assessment. This entry-level “how-to” presentation shows how AI tools were used to simplify test design processes. The presenters share their experience producing a new 50-item multiple choice format placement test for 300 students, measuring listening, grammar, and reading skills. The presenters begin by outlining the motivations for creating the new test, continue by describing prompt engineering techniques used to generate items, and finally discuss how AI and test makers can collaborate ethically and productively. The presenters created prompts for each section, using LLMs (like ChatGPT or Claude). Items created were then human-reviewed and edited. Following a pilot test and prior to implementation, further edits were made with human-AI collaboration. Through trial and error, the new test showed improvement across multiple metrics, including item quality (ID), and test reliability (from α=0.8 to α=0.87). Items and content that did not perform well in-context were easily changed. These results were achieved in a much shorter timeframe than usual (about 50%). This project serves as a blueprint for how departments can reduce development time, increase quality, and subsequently better assess their own target student population.