Requirements-oriented methodology for evaluating ontologies?

Many applications benefit from the use of a suitable ontology but it can be difficult to determine which ontology is best suited to a particular application. Although ontology evaluation techniques are improving as more measures and methodologies are proposed, the literature contains few specific examples of cohesive evaluation activity that links ontologies, applications and their requirements, and measures and methodologies. In this paper, we present ROMEO, a requirements-oriented methodology for evaluating ontologies, and apply it to the task of evaluating the suitability of some general ontologies (variants of sub-domains of the Wikipedia category structure) for supporting browsing in Wikipedia. The ROMEO methodology identifies requirements that an ontology must satisfy, and maps these requirements to evaluation measures. We validate part of this mapping with a task-based evaluation method involving users, and report on our findings from this user study.