On the accuracy of objective image and video quality models: New methodology for performance evaluation

Abstract : There are several standard methods for evaluating the performance of models for objective quality assessment with respect to results of subjective tests. However, all of them suffer from one or more of the following drawbacks: They do not consider the uncertainty in the subjective scores, requiring the models to make certain decision where the correct behavior is not known. They are vulnerable to the quality range of the stimuli in the experiments. In order to compare the models, they require a mapping of predicted values to the subjective scores, thus not comparing the models exactly as they are used in the real scenarios. In this paper, new methodology for objective models performance evaluation is proposed. The method is based on determining the classification abilities of the models considering two scenarios inspired by the real applications. It does not suffer from the previously stated drawbacks and enables to easily evaluate the performance on the data from multiple subjective experiments. Moreover, techniques to determine statistical significance of the performance differences are suggested. The proposed framework is tested on several selected metrics and datasets, showing the ability to provide a complementary information about the models' behavior while being in parallel with other state-of-the-art methods.
Type de document :
Communication dans un congrès
8th International Conference on Quality of Multimedia Experience (QoMEX), Jun 2016, Lisbonne, Portugal. 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX), 2016, 〈10.1109/QoMEX.2016.7498936〉
Liste complète des métadonnées

Littérature citée [18 références]  Voir  Masquer  Télécharger

http://hal.univ-nantes.fr/hal-01395440
Contributeur : Harold Mouchère <>
Soumis le : jeudi 10 novembre 2016 - 16:36:50
Dernière modification le : mardi 23 janvier 2018 - 11:42:04
Document(s) archivé(s) le : mardi 21 mars 2017 - 05:30:32

Fichier

Qomex2016_paper.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

Collections

Citation

Lukáš Krasula, Karel Fliegel, Patrick Le Callet, Miloš Klíma. On the accuracy of objective image and video quality models: New methodology for performance evaluation. 8th International Conference on Quality of Multimedia Experience (QoMEX), Jun 2016, Lisbonne, Portugal. 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX), 2016, 〈10.1109/QoMEX.2016.7498936〉. 〈hal-01395440〉

Partager

Métriques

Consultations de la notice

92

Téléchargements de fichiers

86