Skip to Main content Skip to Navigation
Conference papers

How to benchmark objective quality metrics from paired comparison data?

Abstract : The procedures commonly used to evaluate the performance of objective quality metrics rely on ground truth mean opinion scores and associated confidence intervals, which are usually obtained via direct scaling methods. However, indirect scaling methods, such as the paired comparison method, can also be used to collect ground truth preference scores. Indirect scaling methods have a higher discriminatory power and are gaining popularity, for example in crowdsourcing evaluations. In this paper, we present how the classification errors, an existing analysis tool, can also be used with subjective preference scores. Additionally, we propose a new analysis tool based on the receiver operating characteristic analysis. This tool can be used to further assess the performance of objective metrics based on ground truth preference scores. We provide a MATLAB script with an implementation of the proposed tools and we show one example of application of the proposed tools.
Complete list of metadatas

Cited literature [26 references]  Display  Hide  Download

http://hal.univ-nantes.fr/hal-01395449
Contributor : Harold Mouchère <>
Submitted on : Saturday, December 7, 2019 - 6:50:30 PM
Last modification on : Wednesday, January 15, 2020 - 8:52:23 AM
Long-term archiving on: : Sunday, March 8, 2020 - 12:16:23 PM

File

QoMEX2016.pdf
Publisher files allowed on an open archive

Identifiers

Collections

Citation

Philippe Hanhart, Lukáš Krasula, Patrick Le Callet, Touradj Ebrahimi. How to benchmark objective quality metrics from paired comparison data?. 8th International Conference on Quality of Multimedia Experience (QoMEX), Jun 2016, Lisbon, Portugal. ⟨10.1109/QoMEX.2016.7498960⟩. ⟨hal-01395449⟩

Share

Metrics

Record views

92

Files downloads

176