Pairwise comparisons or constrained optimization?

Abstract

Decision support methodologies provide notations for expressing and communicating the priorities thatinform a decision. Although a substantial literature has explored the theoretical merits of such notationsand methodologies, much less work has investigated their usability in practice, which is of vital importancefor their widespread adoption by users. In this paper, we explore the usability of two well-known prefer-ence elicitation techniques,pairwise comparisonsandconstrained optimization. The techniques were exploredthrough two contrasting crowd worker experiments, a preliminary one evaluatingrecognition, that is, theability to identify the most suitable formulation for a given task, and the othersynthesis, that is, the abilityto construct formulations for a given task. The tasks are based on a case study involving source selection,a well-known problem in the data integration domain. The results of the empirical evaluation show that,overall,pairwise comparisonsresulted in significantly higher performance thanconstrained optimization,yetthere is negligible difference between the usability appraisals for each technique. Furthermore, we observedthat the technique that participants perform better with is not necessarily the one that they consider moreusable.