Skip to Main content Skip to Navigation
Conference papers

How to benchmark objective quality metrics from paired comparison data?

Abstract : The procedures commonly used to evaluate the performance of objective quality metrics rely on ground truth mean opinion scores and associated confidence intervals, which are usually obtained via direct scaling methods. However, indirect scaling methods, such as the paired comparison method, can also be used to collect ground truth preference scores. Indirect scaling methods have a higher discriminatory power and are gaining popularity, for example in crowdsourcing evaluations. In this paper, we present how the classification errors, an existing analysis tool, can also be used with subjective preference scores. Additionally, we propose a new analysis tool based on the receiver operating characteristic analysis. This tool can be used to further assess the performance of objective metrics based on ground truth preference scores. We provide a MATLAB script with an implementation of the proposed tools and we show one example of application of the proposed tools.
Complete list of metadata

Cited literature [26 references]  Display  Hide  Download
Contributor : Harold Mouchère Connect in order to contact the contributor
Submitted on : Saturday, December 7, 2019 - 6:50:30 PM
Last modification on : Tuesday, September 21, 2021 - 4:12:13 PM
Long-term archiving on: : Sunday, March 8, 2020 - 12:16:23 PM


Publisher files allowed on an open archive




Philippe Hanhart, Lukáš Krasula, Patrick Le Callet, Touradj Ebrahimi. How to benchmark objective quality metrics from paired comparison data?. 8th International Conference on Quality of Multimedia Experience (QoMEX), Jun 2016, Lisbon, Portugal. ⟨10.1109/QoMEX.2016.7498960⟩. ⟨hal-01395449⟩



Record views


Files downloads