Skip to Main content Skip to Navigation
Conference papers

On the accuracy of objective image and video quality models: New methodology for performance evaluation

Abstract : There are several standard methods for evaluating the performance of models for objective quality assessment with respect to results of subjective tests. However, all of them suffer from one or more of the following drawbacks: They do not consider the uncertainty in the subjective scores, requiring the models to make certain decision where the correct behavior is not known. They are vulnerable to the quality range of the stimuli in the experiments. In order to compare the models, they require a mapping of predicted values to the subjective scores, thus not comparing the models exactly as they are used in the real scenarios. In this paper, new methodology for objective models performance evaluation is proposed. The method is based on determining the classification abilities of the models considering two scenarios inspired by the real applications. It does not suffer from the previously stated drawbacks and enables to easily evaluate the performance on the data from multiple subjective experiments. Moreover, techniques to determine statistical significance of the performance differences are suggested. The proposed framework is tested on several selected metrics and datasets, showing the ability to provide a complementary information about the models' behavior while being in parallel with other state-of-the-art methods.
Complete list of metadatas

Cited literature [18 references]  Display  Hide  Download

http://hal.univ-nantes.fr/hal-01395440
Contributor : Harold Mouchère <>
Submitted on : Thursday, November 10, 2016 - 4:36:50 PM
Last modification on : Wednesday, January 15, 2020 - 10:26:54 AM
Document(s) archivé(s) le : Tuesday, March 21, 2017 - 5:30:32 AM

File

Qomex2016_paper.pdf
Files produced by the author(s)

Identifiers

Collections

Citation

Lukáš Krasula, Karel Fliegel, Patrick Le Callet, Miloš Klíma. On the accuracy of objective image and video quality models: New methodology for performance evaluation. 8th International Conference on Quality of Multimedia Experience (QoMEX), Jun 2016, Lisbonne, Portugal. ⟨10.1109/QoMEX.2016.7498936⟩. ⟨hal-01395440⟩

Share

Metrics

Record views

539

Files downloads

629