Authentic and concurrent evaluation - Refining an evaluation approach in design science research

This paper addresses the need for more detailed accounts for evaluation in design science research literature. By revisiting a design project regarding the future e-newspaper we give detailed descriptions of its authentic and concurrent evaluation approach by illustrating the what, why and how of all evaluation activities throughout the whole project. The project produced seven different design artifacts that were evaluated. The utility and theoretical outcomes of the evaluation activities clearly influenced design decisions regarding newspaper design, user value and business model design as well as decisions on strategic levels. We emphasize a holistic and concurrent approach to evaluation compared to the general design science research thinking and argue that reflecting on how to seek authenticity is important. By authenticity we refer to the notion of how closely an evaluation captures the context and actual use of an artifact. With the holistic approach we encourage evaluation to be inclusive of different aspects and relationships between stakeholder groups in the evaluation activities. Further we think that concurrency is not narrowed to evaluation but also regards theorizing. While it makes sense for planning to distinguish between phases and stages of evaluation and theorizing, in practice they are intrinsically interlinked and concurrent.