Fast Adaptive Algorithm for Robust Evaluation of Quality of Experience

Transcription

1 Fast Adaptive Algorithm for Robust Evaluation of Quality of Experience Qianqian Xu, Ming Yan, Yuan Yao October 2014

2 1 Motivation Mean Opinion Score vs. Paired Comparisons Crowdsourcing Ranking on Internet 2 Outlier Detection HodgeRank on Graphs LASSO for Outlier Detection Adaptive Least Trimmed Squares 3 Numerical Experiments Simulated Study Real-world Data 4 Conclusions

3 Outline Motivation Outlier Detection Numerical Experiments Subjective Image Quality Assessment Reference Qianqian Xu, Fast-fading Ming Yan, Yuan Yao White-noise iterative Least Trimmed Squares Conclusions

4 Mean Opinion Score Mean opinion score is widely used for evaluation of images, videos, as well books and movies, etc., but Unable to concretely define the concept of scale; Ambiguity interpretations of the scale among users; Difficult to verify whether a participant gives false ratings either intentionally or carelessly.

5 Outline Motivation Outlier Detection Numerical Experiments Paired Comparisons Simpler design with binary choice; Robust decision (invariant up to a monotone transform on personal scaling functions). Which one looks better to you (fastfading vs. white noise)? Qianqian Xu, Ming Yan, Yuan Yao iterative Least Trimmed Squares Conclusions

6 Crowdsourcing Ranking on Internet Start from a movie The Social Network

7 Crowdsourcing Definition The term crowdsourcing is a portmanteau of crowd and outsourcing. It is the act of outsourcing tasks, traditionally performed by an employee or contractor, to an undefined, large group of people or community (a crowd ), through an open call. random participants from Internet; random item pairs in comparison; so, data is incomplete; is imbalanced (heterogeneously distributed); is dynamic; contains outliers.

8 Automatic Outlier Detection in QoE Evaluation iterative Least Trimmed Squares (ilts) is proposed for outlier detection in QoE evaluation. ilts is fast achieves up to 190 times faster than LASSO; ilts is adaptive purifies data automatically without a prior knowledge on the amount of outliers.

9 Least Squares Assume Y α ij = sign(s i s j + Z α ij ) sign( ) = ±1 measures the sign of the value; s = {s 1,, s n} R n is the true scaling score on n items; z α ij is noise. For independent and identically distributed noise zij α with zero mean, the Gauss-Markov Theorem claims Least Square (LS)-rank is the unbiased estimator with minimal variance: 1 minimize s R n 2 (s i s j Yij α ) 2. i,j,α

10 Sparse Outliers If Zij α = Eij α + Nij α, where perturbation E α ij is sparse outliers, LS becomes unstable and may give bad estimation. May due to different test conditions; human errors; abnormal variations in context. How to detect and remove them to achieve a robust estimation against sparse outliers?

11 LASSO 1 minimize s R n,e 2 (s i s j Yij α + Eij α ) 2 + λ E 1. i,j,α Algorithm 1 LASSO for Outlier-Detection and Global Rating Estimation (1) Find the solution path of the Lasso problem; (2) Tuning parameter. Determine an optimal λ by cross-validation with random projections or by looking at the path directly; (3) Rule out outliers and perform least squares to get an unbiased score estimation.

12 Parameter Tuning Remark. Cross-validation may fail when outliers become dense and small in magnitudes. We can look at the solution path directly. An example of the solution path of simulated data with those corresponding to outliers are plotted in red, which mostly lie outside the majority of the paths.

13 Drawbacks of LASSO LASSO is expensive; LASSO needs prior knowledge (i.e., number of outliers in the dataset). Call for an efficient and automatic method for outlier detection!

14 Least Trimmed Squares Use l 0 in the constraint instead of l 1 in the objective. { minimize 1 s R n,e 2 (s i s j Yij α + E α ij )2, i,j,α subject to E 0 K. It is equivalent to minimize s R n,λ subject to Λ α ij (s i s j Yij α)2, (1 Λ α ij ) K, Λα ij {0, 1}. i,j,α i,j,α (1) Here Λ α ij is used to denote the outliers as follows: Λ α ij = { 0, if Y α ij is a outlier, 1, otherwise. (2)

15 Alternating Minimization 1) Fix Λ and update s. We need to solve a least squares problem with the comparisons that Λ α ij = 1. 2) Fix s and update Λ. This time we are solving minimize Λ subject to Λ α ij (s i s j Yij α)2, (1 Λ α ij ) K, Λα ij {0, 1}. i,j,α i,j,α This problem is to choose K elements with largest summation from the set {(s i s j Yij α)2 }. We can choose any Λ such that (1 Λ α ij ) K, Λα ij {0, 1} and i,j,α min i,j,α,λ α=0(s i s j Yij α ) 2 max ij i,j,α,λ α=1(s i s j Yij α ) 2. (4) ij (3)

16 iterative Least Trimmed Squares Algorithm 2 iterative Least Trimmed Squares with K Input: {Yij α }, K 0. Initialization: k = 0, Λ α ij = 1. for k = 1, 2, do Update s k by solving the least squares problem using only the comparisons with Λ α ij = 1. Update Λ k from (4) with one different from previous ones. end for return s.

17 Adaptive Least Trimmed Squares Algorithm 3 Adaptive Least Trimmed Squares Input: {Yij α }, Miter > 0, β 1 < 1, β 2 > 1. Initialization: k = 0, Λ α i,j(k) = 1, K k = 0. for k = 1,, Miter do Update s k with least squares using only the comparisons with Λ α ij (k 1) = 1. Let K k be the total number of comparisons with wrong directions, i.e., Yij α has different sign with si k sj k. { β 1 Kk, if k = 1; K k = min( β 2K k 1, K (5) k ), otherwise. If K k = K k, break. Update Λ(k) using (4) with K = K k. end for Find ŝ with least squares using only the samples with Λ α ij (k) = 1. return ŝ, ˆK = Kk.

18 Remarks β 1 < 1 is small to make sure that the first estimation is underestimated. e.g. β 1 =0.75. β 2 > 1 is small to make sure that it will not overshoot the estimation too much. e.g. β 2 =1.03. Add one step to just compare every pair of two successive items and make the correction on the detection, i.e., if item i is ranking above j but the number of people choosing item i over j is less than the number of people choosing j over i, we can remove those choosing j over i and keep those choosing i over j.

19 Data Description Create a random total order on n candidates as the ground-truth order. Add paired comparison edges (i, j) randomly with preference directions following the ground-truth order. Randomly choose a portion of the comparison edges and reverse them in preference directions. Notation: SN (Sample Number): total number of paired comparisons. ON (Outlier Number): number of outliers. OP (Outlier Percentage): ON/SN.

20 ilts vs. LASSO (Precisions) avg (sd) OP=5% OP=10% OP=15% OP=30% OP=45% OP=50% SN= (0.022) 0.993(0.023) 0.993(0.015) 0.942(0.037) 0.670(0.078) 0.505(0.097) SN= (0.000) 1.000(0.000) 0.998(0.009) 0.976(0.023) 0.751(0.067) 0.503(0.089) SN= (0.000) 1.000(0.000) 1.000(0.000) 0.991(0.013) 0.811(0.060) 0.502(0.090) SN= (0.000) 1.000(0.000) 1.000(0.000) 0.995(0.010) 0.829(0.059) 0.498(0.098) SN= (0.000) 1.000(0.000) 1.000(0.000) 0.998(0.006) 0.847(0.052) 0.499(0.101) Precisions for simulated data via ilts, 100 times repeat. avg (sd) OP=5% OP=10% OP=15% OP=30% OP=45% OP=50% SN= (0.033) 0.962(0.030) 0.958(0.025) 0.905(0.032) 0.698(0.067) 0.513(0.085) SN= (0.011) 0.990(0.014) 0.984(0.016) 0.942(0.022) 0.750(0.056) 0.516(0.084) SN= (0.005) 0.997(0.008) 0.992(0.012) 0.957(0.020) 0.796(0.050) 0.523(0.083) SN= (0.001) 0.999(0.005) 0.996(0.009) 0.970(0.016) 0.818(0.048) 0.518(0.093) SN= (0.002) 1.000(0.000) 0.998(0.006) 0.972(0.016) 0.837(0.038) 0.525(0.088) Precisions for simulated data via LASSO, 100 times repeat.

21 ilts vs. LASSO (Recalls) avg (sd) OP=5% OP=10% OP=15% OP=30% OP=45% OP=50% SN= (0.000) 0.994(0.015) 0.994(0.010) 0.943(0.036) 0.653(0.080) 0.438(0.093) SN= (0.000) 1.000(0.000) 0.999(0.006) 0.978(0.019) 0.727(0.071) 0.456(0.087) SN= (0.000) 1.000(0.000) 1.000(0.000) 0.991(0.012) 0.797(0.062) 0.464(0.089) SN= (0.000) 1.000(0.000) 1.000(0.000) 0.996(0.007) 0.821(0.060) 0.466(0.098) SN= (0.000) 1.000(0.000) 1.000(0.000) 0.998(0.006) 0.842(0.052) 0.470(0.100) Recalls for simulated data via ilts, 100 times repeat. avg (sd) OP=5% OP=10% OP=15% OP=30% OP=45% OP=50% SN= (0.033) 0.962(0.030) 0.958(0.025) 0.905(0.032) 0.698(0.067) 0.513(0.085) SN= (0.011) 0.990(0.014) 0.984(0.016) 0.942(0.022) 0.750(0.056) 0.518(0.084) SN= (0.005) 0.997(0.008) 0.992(0.012) 0.957(0.020) 0.796(0.050) 0.523(0.083) SN= (0.001) 0.999(0.005) 0.996(0.009) 0.970(0.016) 0.818(0.048) 0.518(0.093) SN= (0.002) 1.000(0.000) 0.998(0.006) 0.972(0.016) 0.837(0.038) 0.525(0.088) Recalls for simulated data via LASSO, 100 times repeat.

22 ilts vs. LASSO (F1 scores) avg (sd) OP=5% OP=10% OP=15% OP=30% OP=45% OP=50% SN= (0.012) 0.994(0.019) 0.994(0.012) 0.943(0.036) 0.675(0.079) 0.469(0.095) SN= (0.000) 1.000(0.000) 0.999(0.007) 0.977(0.021) 0.739(0.069) 0.478(0.088) SN= (0.000) 1.000(0.000) 1.000(0.000) 0.991(0.012) 0.804(0.061) 0.482(0.089) SN= (0.000) 1.000(0.000) 1.000(0.000) 0.996(0.009) 0.825(0.059) 0.482(0.098) SN= (0.000) 1.000(0.000) 1.000(0.000) 0.998(0.006) 0.845(0.052) 0.484(0.101) F1 scores for simulated data via ilts, 100 times repeat. avg (sd) OP=5% OP=10% OP=15% OP=30% OP=45% OP=50% SN= (0.033) 0.962(0.030) 0.958(0.025) 0.905(0.032) 0.698(0.067) 0.513(0.085) SN= (0.011) 0.990(0.014) 0.984(0.016) 0.942(0.022) 0.750(0.056) 0.516(0.084) SN= (0.005) 0.997(0.008) 0.992(0.012) 0.957(0.020) 0.796(0.050) 0.523(0.083) SN= (0.001) 0.999(0.005) 0.996(0.009) 0.970(0.016) 0.818(0.048) 0.518(0.093) SN= (0.002) 1.000(0.000) 0.998(0.006) 0.972(0.016) 0.837(0.038) 0.525(0.088) F1 scores for simulated data via LASSO, 100 times repeat.

23 ilts is Fast! time (second) OP=5% OP=10% OP=15% OP=30% OP=45% OP=50% SN= SN= SN= SN= SN= Computing time for 100 runs in total via ilts. time (second) OP=5% OP=10% OP=15% OP=30% OP=45% OP=50% SN= SN= SN= SN= SN= Computing time for 100 runs in total via LASSO. ilts can achieve up to about 190 times faster than LASSO!!!

24 Outline Motivation Outlier Detection Numerical Experiments Conclusions Data Description Figure: Left: PC-VQA dataset; right: PC-IQA dataset. PC-VQA (complete and balanced dataset): 38,400 paired comparisons for 10 reference videos, collected from 209 random observers. PC-IQA (incomplete and imbalanced dataset): 23,097 paired comparisons for 15 reference images, collected from 187 random observers. Qianqian Xu, Ming Yan, Yuan Yao iterative Least Trimmed Squares

25 Experimental Results ID Paired comparison matrixes of reference (a) in PC-VQA dataset. Red pairs are outliers obtained by both ilts and LASSO. Open blue circles are outliers obtained by LASSO but not ilts. Filled blue circles are outliers obtained by ilts but not LASSO.

26 Different rankings after outlier removal Video ID L2 LASSO IRLS 1 1 ( ) 1 ( ) 1 ( ) 9 2 ( ) 2 ( ) 2 ( ) 10 3 ( ) 3 ( ) 3 ( ) 13 4 ( ) 4 ( ) 4 ( ) 7 5 ( ) 5 ( ) 5 ( ) 8 6 ( ) 6 ( ) 6 ( ) 11 7 ( ) 7 ( ) 7 ( ) 14 8 ( ) 8 ( ) 8 ( ) 15 9 ( ) 9 ( ) 9 ( ) 3 10 ( ) 11 ( ) 12 ( ) ( ) 10 ( ) 10 ( ) 4 12 ( ) 12 ( ) 11 ( ) ( ) 13 ( ) 13 ( ) 5 14 ( ) 14 ( ) 14 ( ) 6 15 ( ) 15 ( ) 15 ( ) 2 16 ( ) 16 ( ) 16 ( ) Different rankings for reference (a) in the PC-VQA dataset. The integer represents the ranking position. The number in parentheses represents the global ranking score.

27 Experimental Results ID Paired comparison matrixes of reference (c) in PC-IQA dataset. Red pairs are outliers obtained by both ilts and LASSO. Open blue circles are outliers obtained by LASSO but not ilts. Filled blue circles are outliers obtained by ilts but not LASSO.

28 Different rankings after outliers removal for reference (c) in the PC-IQA dataset Image ID L2 LASSO IRLS 1 1 ( ) 1 ( ) 1 ( ) 8 2 ( ) 2 ( ) 2 ( ) 16 3 ( ) 3 ( ) 3 ( ) 2 4 ( ) 4 ( ) 4 ( ) 3 5 ( ) 5 ( ) 5 ( ) 11 6 ( ) 6 ( ) 7 ( ) 6 7 ( ) 7 ( ) 6 ( ) 12 8 ( ) 8 ( ) 8 ( ) 9 9 ( ) 9 ( ) 9 ( ) ( ) 10 ( ) 10 ( ) 5 11 ( ) 11 ( ) 11 ( ) ( ) 12 ( ) 12 ( ) 7 13 ( ) 13 ( ) 13 ( ) ( ) 14 ( ) 15 ( ) ( ) 15 ( ) 14 ( ) 4 16 ( ) 16 ( ) 16 ( ) Remark: ilts prefers to choose minority votings as outliers, while the LASSO selects the large deviations from the gradient of global ranking score as outliers even when they are in majority votings. Such a small difference only leads to a local order change of nearby ranked items, so both of these two ranking algorithms are stable.

29 Summary ilts is a surprisingly simple, efficient, and automatic algorithm for outlier detection in QoE evaluation. ilts achieves up to 190 times faster than LASSO. ilts can automatically estimate the number of outliers and detect them without any prior information about the number of outliers exist in the dataset. Open resources: report, slides, and codes available on