Sub-corpora impact on system effectiveness

Understanding the factors comprising IR system effectiveness is of primary importance to compare di.erent IR systems. Efiectiveness is traditionally broken down, using ANOVA, into a topic and a system effect but this leaves out a key component of our evaluation paradigm: The collections of documents. We break down effectiveness into topic, system and sub-corpus effects and compare it to the traditional break down, considering what happens when di.erent evaluation measures come into play. We found that sub-corpora are a significant effect. The consideration of which allows us to be more accurate in estimating what systems are significantly di.erent. We also found that the sub-corpora a.ect di.erent evaluation measures in di.erent ways and this may impact on what systems are considered significantly di.erent.