Repository of wisdom: Automated support for composing programming exams

This paper presents a macro-level view around exam composition. While previous work known as BABELnot (Lister et. al. 2012 [1]) developed a micro-level classification scheme to consistently categorise individual exam questions, this paper's contribution uses a more holistic intent towards collective exam composition to build on the past research from the BABELnot project. Specifically, it addresses a higher order, cognitive layer of complexity on top of the exam classification work derived from the BABELnot project to categorise foundation level programming exam questions. In preparation for this, we analysed use cases for a programming questions database for the composition of exams and selected two for further analysis and implementation. A database designed for use by both educators and researchers exists, called "The repository of Wisdom" (RoW), however, it needs further enhancements to support the goals of this paper. The RoW was designed and implemented as part of previous work (Hamilton, D'Souza, Harland, Rosalina 2014 [3]) that classified questions in programming exams. The retrieval of these questions can be by various attributes such as topic, concept, aptitude, content, level of the course and benchmarked results, with interesting and innovative retrieval options related to ranked queries. The selection process can also be influenced by difficulty scores, ratings and comments given by the instructors who submitted the questions or others who may have trialled them. We would like the repository to be further evaluated in the "real world" by computer scientists and, in particular, academics assessing novice programming ability or designing entry level exams. We have built ontologies and mechanisms for storing and retrieving exam questions and also discuss these in this paper.