Daylight Discomfort Glare Evaluation with Evalglare: Influence of Parameters and Methods on the Accuracy of Discomfort Glare Prediction

Transcription

1 buildings Article Daylight Discomfort Glare Evaluation with Eval: Influence Parameters Methods on Accuracy Discomfort Glare Prediction Clotilde Pierson 1, * ID, Jan Wienold 2 Magali Bodart 1 1 Université catholique de Louvain, 1348 Louvain-la-Neuve, Belgium; magali.bodart@uclouvain.be 2 École Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerl; jan.wienold@epfl.ch * Correspondence: clotilde.pierson@uclouvain.be; Tel.: Received: 26 June 2018; Accepted: 17 July 2018; Published: 24 July 2018 Abstract: Nowadays, discomfort indices are frequently calculated by using eval. Due to lack knowledge on implications methods parameters eval, default settings are ten used. But wrong parameter settings can lead to inappropriate source detection refore to invalid indices calculations erroneous classifications. For that reason, this study aims to assess influence several source detection methods parameters on accuracy discomfort prediction for daylight. This analysis uses two datasets, representative two types discomfort : saturation contrast. By computing three different statistical indicators to describe accuracy discomfort prediction, 63 different settings are compared. The results suggest that choice an eval method should be done when considering type that is most likely to occur in visual scene: task area method should be preferred for contrast scenes, threshold method for saturation scenes. The parameters that should be favored or avoided are also discussed, although a deeper understing discomfort mechanism a clear definition a source would be necessary to reliably interpret se results. Keywords: eval; discomfort ; source detection; evaluation; parametric analysis 1. Introduction According to International Commission on Illumination (CIE), is defined as condition vision in which re is discomfort [discomfort ] or a reduction in ability to see significant objects [disability ], or both, due to an unsuitable distribution or range luminances or to extreme contrasts in space or time [1]. When studying discomfort nowadays, most common method is to compare subjective ratings that are made on discomfort scales, values discomfort indices calculated from measured physical quantities. There are four main physical quantities, on which discomfort indices are based [1]: luminance source(s) in field view, solid angle source(s), luminance background, position index source(s). To measure se quantities efficiently in situ, High Dynamic Range (HDR) imaging technique is used to create a 180 luminance map containing subject s field view. Discomfort indices are n calculated afterward on basis se luminance maps. Alternatively, such 180 luminance maps can also be generated through simulation tools. The calculation discomfort indices based on a luminance map is refore an essential step an inherent part discomfort studies. The difficulty this step, however, lies in identification source(s) in field view. If a source has been identified in 180 luminance map, its solid angle, position index, Buildings 2018, 8, 94; doi: /buildings

2 Buildings 2018, 8, luminance can be easily determined. The background luminance can also be quickly evaluated as average luminance luminance map, excluding source. But, as reported in [2], definition source is still ambiguous, re is no universal method identifying one. The most common methods that are currently used are ones implemented in eval. Eval is a Radiance-based tool developed by J. Wienold as part a larger study with J. Christfersen [3], during which y created Daylight Glare Probability (DGP) index. This tool requires a 180 luminance map containing field view as input, n runs an algorithm to detect sources, finally calculates different discomfort indices from detected sources [4 6]. The output eval is refore a list discomfort indices values that are calculated on basis sources detected by algorithm in luminance map. Eval is a powerful tool that makes discomfort indices calculations easier quicker, which is reason why it is so widely used. Eval algorithm enables use three different methods for source detection: factor method, threshold method, task area method, which will be explained later. The task area method is one that provided best results for Wienold s study [7] in terms correlation with subjective ratings. However, factor method is more widely used, as a consequence being default method implemented in eval. What is more, factor method is also default method when performing a point-in-time analysis in daylight-simulation stware DIVA-for-Rhino [8], re exists no option in user-friendly interface to modify it. Moreover, several options parameters can be changed in eval, which modify source detection algorithm. But, se options parameters are seldom used as compared to default or recommended ones, probably because lack knowledge about ir influence on sources detection. Surprisingly, eval tool has never been object a validation study or than Wienold Christfersen s study although it has been now used for over 10 years. There exist no recommendations based on scientific evidence on which methods should be preferred in which cases, which options should be applied for which conditions. Therefore, aim this study is to investigate effect different eval methods parameters on discomfort prediction. Although it would be interesting to look at sensitivity discomfort indices values when varying eval methods parameters, focus this study is about determining which method parameters fer best discomfort prediction in comparison with subjective ratings. Two datasets are used in this study, comprising, for each evaluation, a subjective rating a 180 luminance map containing subject s field view. Each dataset is representative a discomfort type, i.e., contrast or saturation. By varying eval options combination eval methods parameters when calculating discomfort indices based on luminance maps, options that fer best prediction subjective ratings for two types may be identified. 2. Datasets Two datasets containing daylight discomfort evaluations are used for this study. The demographic information se datasets is available in Table 1. Table 1. Demographic general information field laboratory datasets. Experiment Subjects Men/Women Ratio (%) Mean Age (SD) Glare Evaluations Field study 82 43/57 36 (11) 141 Laboratory study 41 73/27 26 (3) Point Glare Scale No discomfort A small discomfort A moderate discomfort A large discomfort Imperceptible Noticeable Disturbing Intolerable Glare Rating Ratio (%)

3 Buildings 2018, 8, The first dataset was collected during a field study that was conducted in Louvain-la-Neuve (Belgium) from June to August subjects participated in study, which consisted collecting almost simultaneously subjective ratings 180 luminance maps containing subject s field view in ir real fice. The study took place in several university buildings, for fice desks that are located next to an east-, west-, or south-oriented window. Each subject did two evaluations first one for situation y were working in, second one for situation with opened shading devices artificial lighting turned f resulting in a total 164 evaluations. Despite fact that field study was conducted under clear sky conditions, 23 evaluations had to be disregarded due to unstable lighting conditions. The subjective ratings were preceded by a short visual task, were made on a 4-point scale, on which subject has to evaluate his/her perception discomfort due to as eir No discomfort, A small discomfort, A moderate discomfort, or A large discomfort. The detailed experiment protocol has been published in [9]. The second dataset was collected during a laboratory study that was conducted in two fice-like cells in Freiburg (Germany) in spring/summer/autumn months In total, 150 subjects had to perform five visual tasks, each followed by a discomfort rating, for maximum four different visual scenes shading device was varied at two different times day morning afternoon. However, only data from 41 subjects from one five visual tasks, namely typing task, collected in spring summer months were available used in this study, resulting in a total 180 evaluations. The laboratory study was conducted under clear sky conditions, fice-like cells were rotated so that sun direction was perpendicular to façade. The discomfort rating was done on a different 4-point scale, on which subject was asked to evaluate his/her perception as Imperceptible, Noticeable, Disturbing, or Intolerable. Simultaneously with discomfort rating, a 180 luminance map was created with luminance camera positioned next to subject s head, at eye level. More details about this laboratory study have been published in [10]. The choice using two different datasets was made so that a greater variety visual scenes is studied, especially two types discomfort, namely contrast saturation. The lighting conditions two studies are very different, as can be seen in 1. Vertical illuminance at eye level is much higher in case laboratory study, with a median illuminance level above 2000 lux, when compared to a median illuminance level around 500 lux for field study. This implies that when discomfort was perceived in field study, discomfort most probably was caused from high contrast source background. On opposite, when discomfort was perceived in laboratory study, discomfort was most probably due to a too bright visual scene. Therefore, most discomfort situations in field study were caused by contrast, whereas discomfort situations in laboratory study were mainly caused by saturation. These two datasets can refore be used as representatives two main types discomfort. Moreover, two studies have several differences in ir experimental design, besides being a field or laboratory study type. One main differences is use two different rating scales for subjective evaluation discomfort (Table 1). Anor main difference is that, in field study, HDR image is captured directly after subjective rating, but at exact same place subject s eyes, while in laboratory study, HDR image is captured at exact same time than subjective rating, but next to subject s head. These differences make two datasets representative many discomfort studies, y enable results to be relevant for a large number future studies. These differences also hinder comparison accuracy discomfort prediction datasets, which is not subject this study.

4 Buildings 2018, 8, Buildings 2018, 8, x FOR PEER REVIEW Boxplots 1. Boxplots vertical vertical illuminance illuminance at eye at eye level level field field study study laboratory laboratory study. study. 3. Methodology 3. Methodology When using eval, choice a method parameters influence source detection in a luminance When map. using The eval, detected choice sources, a method in turn, influence parameters discomfort influence indices source detection calculation. Atin last, a luminance accuracymap. discomfort The detected prediction sources, varies in turn, according influence to se discomfort discomfort indices indices. To calculation. determine which At last, method accuracy which discomfort parameters prediction eval are varies according most appropriate to se discomfort each type discomfort indices. To research, determine which reported method effects which choice parameters a method eval parameters are in eval most appropriate for each type discomfort research, reported effects choice a method on accuracy discomfort prediction is investigated. However, effects this choice on parameters in eval on accuracy discomfort prediction is investigated. However, in- steps, namely source detection discomfort indices calculation, effects this choice on in- steps, namely source detection discomfort will not be examined here. indices calculation, will not be examined here. 63 different eval options each option is a different combination one three eval 63 different eval options each option is a different combination one three methods different parameters for detecting sources in a luminance map are tested. eval methods different parameters for detecting sources in a luminance map are The tested. accuracy The accuracy discomfort discomfort prediction prediction se 63 se different 63 different options options is evaluated is evaluated through through three different three different statistical statistical approaches approaches for eachfor each two datasets. two datasets. The 63The studied 63 studied eval eval options, options, as well as as well three as statistical three statistical approaches approaches are explained are explained Sections in Sections Studied Studied Eval Eval Methods Methods Parameters Parameters The The three three different different methods implemented in evalare, are, as asnamed namedin inthis this article, article, factor factor method, method, threshold method, task area method. The The algorithm factor method detectsin in 180 luminance map map all all pixels pixels having having a a luminance value higher than mean luminance luminance map map multiplied by aby certain a certain factor. These factor. pixels These arepixels treatedare astreated as sources. Insources. this study, In this 4 different study, 4 factors different were factors tested: were a multiplying tested: a factor multiplying 5 (b5), factor which is 5 (b5), default which is method default in eval, method in 6 (b6), eval, 7 (b7), 6 (b6), 87 (b8). (b7), 8 (b8). The The algorithm threshold method detectsin in 180 luminance map map all pixels all pixels having having a a luminance value thatis is higher thana a certain threshold, treat treat se se pixels pixels as as sources. sources. In In literature, several thresholdshave havebeen beenused used throughout years years [11 15], [11 15], but but three three most most common common ones ones are are a threshold a threshold cd/m 22 (b1000), 2000 cd/m 22 (b2000), cd/m cd/m 2 (b4000). 2 (b4000).

5 Buildings 2018, 8, Buildings 2018, 8, x FOR PEER REVIEW 5 32 The algorithm task area method detects in 180 luminance map all pixels having a luminance The algorithm value higher task thanarea method mean luminance detects 180 a defined luminance taskmap areaall inpixels having luminance a map luminance multiplied value byhigher a certain than factor. mean This luminance last method a defined requirestask prior area definition luminance a taskmap area. The multiplied recommended by a multiplying certain factor. factor This for last method task area requires method prior in eval definition is 5. Three a task or area. factors The will recommended also be tested multiplying in this study, factor resulting for in task multiplying area method factors in eval 3 (t3), is 4 5. (t4), Three 5 (t5), or factors 6 (t6). will also be tested in this study, resulting in multiplying factors 3 (t3), 4 (t4), 5 (t5), 6 (t6). Four or eval parameters are investigated in this study. First, background luminance Four or eval parameters are investigated in this study. First, background luminance definition will be tested, since re exist two different definitions this background luminance. definition will be tested, since re exist two different definitions this background luminance. The The first definition is one that is provided by CIE, which approximates background first definition is one that is provided by CIE, which approximates background luminance luminance as indirect vertical illuminance visual field divided by π (_def). The second one is as indirect vertical illuminance visual field divided by π (_def). The second one is mamatical definition, i.e., mamatical average luminance values in luminance mamatical definition, i.e., mamatical average luminance values in luminance map map excluding excluding sources sources (_Lb). (_Lb). Secondly, Secondly, search search radius, radius, which which is is distance distance within within which which two two glaring glaring pixels pixels are are included included in same same source, source, will will be varied. be varied. The The default default value value search radius, search 0.2 radius, (_def), 0.2 as well (_def), as one as well smaller as one value smaller 0.06 value (_r0.06) 0.06 (_r0.06) one larger value one larger 0.3 value (_r0.3) will 0.3 be(_r0.3) will tested. The task be tested. area size, The task which area issize, onlywhich required is only when required usingwhen task using area method, task area will method, n be varied will n as well. be varied The task well. areathe is defined task area asis defined area where as area subject s where eyes subject s are focused. eyes are focused. The usual task The area usual sizetask hasarea an opening size has angle opening around angle 60 (_def) around to60 correspond (_def) to correspond to ergorama, to ergorama, i.e., area around i.e., area computer around screen computer keyboard. screen A smaller keyboard. task area A smaller size 30 task (_ta0.52) including area size 30 (_ta0.52) only computer including screen, only computer a larger size screen, 90 (_ta1.57) will a larger size 90 also(_ta1.57) will be tested. At also last, be tested. smooth At parameter, last, which smooth should parameter, mainly be which usedshould when mainly visualbe scene used includes when blinds, visual investigated. scene includes Thisblinds, parameter is includes investigated. non-glaring This parameter pixels (pixels includes blind non-glaring slats forpixels instance) (pixels nested blind aslats for source instance) (a nested source in seen a through source (a blinds for source instance) seen through to blinds source, for smoothing instance) to source source, area. smoothing The accuracy discomfort source area. prediction The accuracy will thus discomfort be tested with prediction (_sm) without will thus (_def) be tested smooth with (_sm) parameter. without Table 2 (_def) summarizes smooth all 63 parameter. eval options that were tested in this study provides an example differences Table 2 summarizes in sources all 63 eval detection options that se were options. tested in The this colored study parts provides in images an example differences in sources detection se options. The colored parts in correspond to different detected sources. The choice colors is romly made by eval images correspond to different detected sources. The choice colors is romly made by algorithm has no signification. eval algorithm has no signification. Table 2. Example a visual scene from lab study differences in source detection for all Table 2. Example a visual scene from lab study differences in source detection for all eval options tested. eval options tested. def r0.06 r0.3 Lb sm sm ta0.52 ta1.57 b5 b5 / / b6 b6 / / b7 b7 / / b8 / / b1000 b1000 / /

6 Buildings 2018, 8, Table 2. Cont. Buildings 2018, 8, x FOR PEER REVIEW 6 33 def r0.06 r0.3 Lb sm ta0.52 ta1.57 b2000 b2000 / / b4000 b4000 / / t3 t4 t4 t5 t5 t Statistical Approaches 3.2. Statistical Approaches Instead Instead looking looking at at direct direct effects 63 63different differenteval eval options options on on source source detection detection or or on on discomfort indices calculation, reportedeffects effects on on accuracy accuracy discomfort discomfort prediction is is studied. The accuracy discomfort prediction can can be defined be defined as how as well howa well a discomfort index can predict corresponding subjective rating, rating, i.e., i.e., how how well well this this index index correlates with with rating made by subjects. Since Since aim aim this this study is is to produce results that are widely interpretable, it was it was decided decided not not to choose to choose only only one one daylight daylight discomfort index, index, but but to to base base statistical analyses on on five five most commonly most commonly used daylight used daylight discomfort discomfort indices, indices, namely namely Daylight Daylight Glare Glare Probability (DGP) [3], [3], Discomfort Discomfort Glare Glare Index Index (DGI) (DGI) [16], [16], CIE Glare CIE Glare Index Index (CGI) (CGI) [17], [17], modified modified Discomfort Discomfort Glare Glare Index (DGImod) [18], Unified Glare Probability (UGP) [19]. These discomfort Index (DGImod) [18], Unified Glare Probability (UGP) [19]. These discomfort indices are indices are calculated with each 63 eval options, for both datasets. The gr means, calculated with each 63 eval options, for both datasets. The gr means, i.e., means i.e., means throughout 63 eval options mean indices values, are reported in throughout 63 eval options mean indices values, are reported in Table 3. Table 3. Table 3. Gr mean each discomfort index value for both datasets. Table 3. Gr mean each discomfort index value for both datasets. Experiment DGP DGP DGI DGI CGI DGImodUGP UGP Field Field study study Laboratory study Laboratory study Three statistical approaches have been chosen to compare accuracy discomfort prediction Three statistical resulting from approaches 63 eval have been options. chosen For each to compare approach, an accuracy indicator discomfort relationship prediction resulting subjective from 63 ratings eval each options. For five each discomfort approach, anindices indicator calculated relationship with 63 eval subjective options is evaluated. ratings These indicators each are five discomfort Spearman correlation indices coefficient, calculated Area with 63 eval Under options Curve (AUC) is evaluated. a binary Theselogistic indicators regression, are Spearman corrected correlation Akaike s coefficient, Information Area Criterion (AICc) an ordinal logistic regression. Since 63 eval options are compared in this

7 Buildings 2018, 8, Under Curve (AUC) a binary logistic regression, corrected Akaike s Information Criterion (AICc) an ordinal logistic regression. Since 63 eval options are compared in this study, p-values Spearman test logistic regressions are adjusted with a Bonferroni correction. This correction, which can be applied without any distributional assumptions, reduces risk a type I error, i.e., getting a significant result due to chance. These indicators are n compared 63 eval options. This comparison is first made visually through boxplots, n statistically through significance testing or threshold checking. Each five discomfort indices is considered in comparison, which is done separately for two datasets Spearman Correlation Coefficient Buildings 2018, 8, x FOR PEER REVIEW 7 32 The first risk statistical a type approach I error, i.e., isgetting baseda on significant Spearman result correlation. due to chance. Spearman These indicators correlation are n was chosen compared 63 eval options. This comparison is first made visually through boxplots, over Pearson correlation because one two variables 4-point scale is an ordinal n statistically through significance testing or threshold checking. Each five discomfort variable, indices Spearman is considered correlation in comparison, uses which rank is done separately data points for instead two datasets. ir value [20]. The correlation coefficient, with ir Bonferroni adjusted p-values, were evaluated each Spearman Correlation Coefficient five discomfort indices calculated with 63 eval options, subjective ratings made on 4-point The first statistical rating approach scaleis for based both on Spearman datasets. correlation. The Spearman correlation was coefficient chosen is used over Pearson correlation because one two variables 4-point scale is an ordinal as first indicator accuracy discomfort prediction: larger coefficient, variable, Spearman correlation uses rank data points instead ir value [20]. The better prediction. correlation coefficient, In literature, with ir Bonferroni absoluteadjusted thresholds p-values, have were been evaluated used to define each magnitude a correlation five coefficient: discomfort a coefficient indices calculated 0.2with is considered 63 eval to be options, a practically subjective significant ratings effect, made one 0.5 a moderate on effect, 4-point onerating 0.8scale strong for both effect datasets. [21]. The Spearman correlation coefficient is used as first indicator accuracy discomfort prediction: larger coefficient, better To determine wher difference two correlation coefficients is statistically significant, prediction. In literature, absolute thresholds have been used to define magnitude a several significance correlation coefficient: tests exist. a coefficient In this study, 0.2 is considered cocor package to be a practically in R [22] significant was used effect, to one apply different significance a moderate tests. These effect, tests one are designed 0.8 a strong toeffect compare [21]. two correlation coefficients based on dependent groups ( two To correlation determine wher coefficients difference are evaluated ontwo correlation same dataset) coefficients with overlapping is statistically variables significant, several significance tests exist. In this study, cocor package in R [22] was used to apply (one two variables correlation is same). The significant difference score two 10 different significance tests. These tests are designed to compare two correlation coefficients based coefficients, on i.e., dependent groups two( eval two correlation options, coefficients is givenare oneevaluated point ifon result same dataset) test with comparing se two coefficients overlapping variables is found(one to be statistically two variables significant. correlation The significance is same). The difference significant each eval difference option score is n evaluated two coefficients, on this i.e., score, going two from eval 0 toptions, 50, since is given 10 significance one point if tests are result test comparing se two coefficients is found to be statistically significant. The performed for each five discomfort indices. significance difference each eval option is n evaluated on this score, going from 0 to 50, since 10 significance tests are performed for each five discomfort indices Area Under Receiver Operating Characteristic (ROC) Curve Binomial Logistic Regression Models Area Under Receiver Operating Characteristic (ROC) Curve Binomial Logistic Regression Models The second statistical approach requires computation binary logistic regression models, The second statistical approach requires computation binary logistic regression models, which use a binary variable as dependent variable. Therefore, two 4-point scales are first which use a binary variable as dependent variable. Therefore, two 4-point scales are first transformed transformed into binary into binary scales: scales: no discomfort-discomfort no discomfort-discomfort ( ( 2). 2). 2. Binary transformation (no discomfort discomfort) subjective scales used in 2. Binary field transformation laboratory studies. (no discomfort discomfort) subjective scales used in field laboratory studies. Binary logistic regression models are n computed, with binary discomfort rating as dependent variable, each one five discomfort indices calculated with each one 63 eval options as independent variable. The binary logistic regression models estimate probability that discomfort due to is perceived given value explanatory variable, i.e., discomfort index. Bonferroni correction is applied when checking significance each logistic regression model. Binary logistic regression models are n computed, with binary discomfort rating as dependent variable, each one five discomfort indices calculated with each one

8 Buildings 2018, 8, x FOR PEER REVIEW 8 32 Buildings 2018, 8, To compare 63 eval options, Area Under Receiver Operating Characteristic (ROC) Curve (AUC), specific to each binary logistic regression model, is used. The ROC curve a 63logistic eval regression options model as illustrates independent its diagnostic variable. ability Thefor binary each possible logisticdiscrimination regression models threshold. estimate probability In or words, that discomfort ROC curve due can tobe plotted is perceived in such a way given that each value point curve explanatory represents variable, a i.e., different discomfort cut-f value, index. Bonferroni x y coordinates correction isthis applied point correspond, when checking respectively, significance to False each logistic Positive regression Rate (FPR) model. True Positive Rate (TPR) model when using this cut-f value To ( compare 3). The AUC 63 eval a logistic options, regression Area model Under is area Receiver comprised Operating beneath Characteristic ROC curve (ROC) model. The larger AUC, better model, as probability good detection (TPR) is Curve (AUC), specific to each binary logistic regression model, is used. The ROC curve a logistic maximized, whereas probability false alarm (FPR) is minimized. The AUC each binary regression model illustrates its diagnostic ability for each possible discrimination threshold. In or logistic regression model is used as second indicator accuracy discomfort words, ROC curve can be plotted in such a way that each point curve represents a different prediction. In literature, absolute thresholds for AUC are used to define quality cut-f discrimination value, a xmodel: yan coordinates AUC < 0.6 corresponds this point to a correspond, failing model, respectively, an AUC 0.6 to to a poor False model, Positive Rate an (FPR) AUC 0.7 to an True acceptable Positivemodel, Rate (TPR) an AUC model 0.8 to an when excellent using model this[23,24]. cut-f value ( 3). The AUCSimilarly a logistic to regression comparison model Spearman is areacorrelation comprised coefficients, beneath significance ROC curve tests can be model. The larger applied to AUC, determine better wher model, difference as probability two AUC good is statistically detection significant. (TPR) is maximized, Three whereas different probability significance tests false [25 27] alarm were (FPR) applied is minimized. this study The through AUC each proc binary package logistic in R. These regression model tests is used are designed as second to compare indicator AUC accuracy two paired ROC discomfort curves, namely prediction. ROC curves In logistic literature, regression models built on two variables from same dataset. Using same scoring system as absolute thresholds for AUC are used to define quality discrimination a model: an AUC for Spearman indicator, significance difference each eval option resulting < 0.6 corresponds to a failing model, an AUC 0.6 to a poor model, an AUC 0.7 to an acceptable from AUC approach is evaluated on a score going from 0 to 15, since three significance tests are model, an AUC 0.8 to an excellent model [23,24]. performed for each five discomfort indices. 3. Graph Receiver Operating Characteristic (ROC) curve corresponding to binary 3. Graph Receiver Operating Characteristic (ROC) curve corresponding to binary logistic regression model computed for Daylight Glare Probability (DGP) index calculated with logistic regression model computed for Daylight Glare Probability (DGP) index calculated with t4_def eval option on field study dataset (AUC = ). t4_def eval option on field study dataset (AUC = ) Corrected Akaike s Information Criterion Ordinal Logistic Regression Models Similarly The last to statistical comparison approach Spearman involves correlation computation coefficients, ordinal significance logistic regression tests canmodels. be applied to determine Ordinal logistic wher regression difference models can interpreted two AUC as an extension is statistically binary significant. logistic Three regression different significance models, tests which [25 27] allows were for more applied than in two this study response through categories. proc These package models can in R. refore These tests be are designed computed to compare directly using AUC a 4-point two paired scale ROC as curves, dependent namely variable. ROC curves Instead logistic estimating regression models probability built onthat twodiscomfort variablesdue fromto same will be dataset. perceived, Using ordinal same logistic scoring regression systemmodels as for Spearman measure indicator, how likely significance response is to fall difference below one three each thresholds eval option 4-point resulting scale, given from value independent variable, i.e., discomfort index. Ordinal logistic regression AUC approach is evaluated on a score going from 0 to 15, since three significance tests are performed for each five discomfort indices Corrected Akaike s Information Criterion Ordinal Logistic Regression Models The last statistical approach involves computation ordinal logistic regression models. Ordinal logistic regression models can be interpreted as an extension binary logistic regression

9 Buildings 2018, 8, models, which allows for more than two response categories. These models can refore be computed directly using a 4-point scale as dependent variable. Instead estimating probability that discomfort due to will be perceived, ordinal logistic regression models measure how likely response is to fall below one three thresholds 4-point scale, given value independent variable, i.e., discomfort index. Ordinal logistic regression models are computed for each one five discomfort indices that were calculated with each one 63 eval options. Bonferroni correction is applied when checking significance each model. By comparing how well ordinal models fit to data, 63 eval methods can be compared, so that eval option providing best accuracy discomfort prediction can be highlighted. One common way comparing fit ordinal logistic regression models is to evaluate ir corrected Akaike s Information Criterion (AICc). The AICc is used as an indicator relative information that is lost when a given model is chosen over anor model developed on same dataset. Therefore, AICc can estimate relative quality statistical models computed on same dataset, but does not give any information about absolute quality model. As a result, AICc thresholds indicating absolute quality a model cannot be found in literature. However, considering each dataset separately, lower AICc value, better model. The AICc each ordinal logistic regression model is used as third last indicator accuracy discomfort prediction to compare 63 eval options. In contrast to Spearman AUC statistical approaches, re exists no significance test to statistically compare AICc two ordinal logistic regression models. But, several thresholds difference AICc two models that were computed from same dataset have been published in literature [28 31]. For this study, a threshold 10 points, namely a difference 10 points AICc values two models, was used to determine that model with lower AICc value is significantly better than or one. If difference AICc values two models that were computed from same dataset is larger or equal to 10, significant difference score se two models is given one point. The significance difference two eval options is evaluated on a score from 0 to 5 for each dataset, since AICc difference test is done for each five discomfort indices. 4. Results Discussion In this section, results study are presented discussed. First, three eval methods are compared, followed by comparison two background luminance definitions, three search radius, three task area sizes. At last, use or not smooth parameter is investigated as well. Six graphs are used for each comparison: on left, three graphs related to field study, on right, three graphs related to laboratory study. From top to bottom, first graph uses results first statistical approach, namely Spearman correlation; second graph uses results second statistical approach, namely AUC binary logistic regression models;, third graph uses results third statistical approach, namely AICc ordinal logistic regression models. The statistical results Spearman correlation logistic regressions that were found to be non-significant with Bonferroni correction are still included in graphs, but y can be recognized by graphical means. The datapoints based on DGP index were removed from boxplot graphs. The DGP model is indeed partly based on vertical illuminance, which does not depend on detected sources. These datapoints tend refore to remain stable throughout different eval options prevent us from drawing observations from graphs. However, se datapoints are still plotted in boxplot graphs as small orange triangle points. Finally, tables summarize for each comparison total significant difference scores, which are weighted sum significant difference scores three statistical approaches (Equation (1)): S SD = 1 3 S SD,Spearman S SD,AUC S SD,AICc 5 (1)

10 4.1. Eval Methods 4.1. Eval Methods In s 4 9, indicators accuracy discomfort prediction are compared In s 4 9, indicators accuracy discomfort prediction are compared 11 different eval methods used in this study factor method with factor 5, 6, 7, 8; 11 different eval methods used in this study factor method with factor 5, 6, 7, 8; threshold Buildings 2018, method 8, 94 with threshold 1000, 2000, 4000; task area method with factor 3, 4, 5, All threshold method with threshold 1000, 2000, 4000; task area method with factor 3, 4, 5, 6. All or parameters (background luminance definition, search radius, task area size, smooth or parameters (background luminance definition, search radius, task area size, smooth option) with Sare SD, set total to ir significant default difference value, so score; that methods can be compared on neutral basis. This option) set ir default value, so that methods can be compared on a neutral basis. This comparison S is SD,Spearman, done significant for each difference five score discomfort from Spearman indices statistical (DGP, approach; DGI, CGI, DGImod, UGP), comparison is done for each five discomfort indices (DGP, DGI, CGI, DGImod, UGP), S SD,AUC for both, significant field (left) difference score laboratory from (right) AUC studies. statisticalnon-significant approach;, statistics (Spearman for both field (left) laboratory (right) studies. Non-significant statistics (Spearman correlation S SD,AICc, significant logistic regressions) difference score are from included AICc in statistical graphs but approach. are plotted as dimmed dots. The correlation logistic regressions) are included in graphs but are plotted as dimmed dots. The methods are classified along X-axis in an ascending order performance, in such way that methods 4.1. Eval are classified Methodsalong X-axis in an ascending order performance, in such a way that best performing method can be easily recognized on right. best performing method can be easily recognized on right. From In s graphs 4 9, (s indicators 4 9), it accuracy appears that discomfort task area method prediction seems are compared to be more appropriate From graphs (s 4 9), it appears that task area method seems to be more appropriate for 11 different field study, eval methods threshold used inmethod this study factor for laboratory method with study. factor The 5, 6, task 7, area 8; used threshold with for field study, threshold method for laboratory study. The task area used with a factor method (t4) with threshold threshold 1000, 2000, 2000 cd/m 4000; 2 (b2000) task area method methods with factor providing 3, 4, 5, best 6. accuracy All or factor 4 (t4) threshold 2000 cd/m discomfort parameters (background prediction luminance throughout definition, 2 (b2000) are methods providing best accuracy three search different radius, statistical task areaapproaches. size, smooth In option) graphs, are discomfort prediction throughout three different statistical approaches. In graphs, non-significant set to ir default statistics Spearman value, so that methods correlation can becoefficients compared onor a neutral logistic basis. regression This comparison models are is non-significant statistics Spearman correlation coefficients or logistic regression models are identifiable done for each as transparent five discomfort points. Only indices few (DGP, appear DGI, to CGI, be non-significant, DGImod, UGP), se for both are only identifiable as transparent points. Only a few appear to be non-significant, se are only found field in (left) factor laboratory method. (right) For both studies. studies, Non-significant factor methods statistics (b5, (Spearman b6, b7, correlation b8) are ones logistic that are found producing regressions) in factor are worst included method. discomfort in For both graphs studies, prediction. but are plotted factor In addition, as methods dimmed (b5, dots. b6, differences The b7, b8) methods are are ones classified that are methods producing are along more pronounced X-axis worst in iscomfort ascending laboratory order prediction. study performance, In addition, than for in such field a waydifferences study. thatthis best is probably performing partly method methods due to are can more noise be easily pronounced inherent recognized for to field on studies laboratory right. study than for field study. This is probably partly due to data. noise inherent to field studies data. 4. Comparison rho for studied eval Comparison Comparison Spearman Spearman rho rho for for each each discomfort discomfort index index studied studied eval eval methods methods (field (field study). methods (field study). 5. Comparison Spearman rho for each discomfort index studied eval methods Comparison Spearman rho for each discomfort index studied eval Comparison study). Spearman rho for each discomfort index studied eval methods methods (laboratory (laboratorystudy).

11 Buildings 2018, 8, 94 Buildings 2018, 8, x FOR PEER REVIEW Buildings 2018, 8, x FOR PEER REVIEW Buildings 2018, 8, x FOR PEER REVIEW ComparisonArea AreaUnder Under Curve Curve (AUC) (AUC) for Comparison for each eachdiscomfort discomfort index index Comparison Area Under Curve (AUC) for each discomfort index studied6.eval methods (field study). studied eval methods (field study). studied eval methods(field (fieldstudy). study). 7. Comparison AUC for each discomfort index studied eval methods 7. Comparison AUC for each discomfort index studied eval methods (laboratory study). 7. Comparison Comparison AUC AUC for for each each discomfort discomfort index index studied studied eval eval methods methods 7. (laboratory study). (laboratory study). (laboratory study). 8. Comparison Akaike s Information Criterion (AICc) for each discomfort index Comparison Akaike s Information Criterion (AICc) for each discomfort index 8. studied eval methods (field study). studied eval methods (field study). 8.8.Comparison Comparison Akaike s Information Criterion (AICc) fordiscomfort each discomfort index Akaike s Information Criterion (AICc) for each index studied eval methods (field study). studied eval methods (field study).

12 Buildings 2018, 8, Buildings 2018, 8, x FOR PEER REVIEW Comparison AICc for each discomfort index studied eval methods 9. Comparison AICc for each discomfort index studied eval methods (laboratory study). (laboratory study). As observed in Tables 4 5, significant difference scores do not exceed 82%. This is due to From fact that graphs difference (s 4 9), ittwo appears eval that options task used area method to derive seems DGP to be will more almost appropriate never forbe large field study, significant, since threshold this index method was for designed laboratory be robust study. against The task variations area used with asource factor 4 (t4) detection threshold methods. Therefore, 2000 cd/m a score 2 (b2000) 80% are means methods that for all providing or discomfort best accuracy indices discomfort each prediction significance throughout test applied, three difference different statistical approaches. eval Inmethods graphs, were significant. non-significant As statistics Spearman already observed from correlation graphs, coefficients differences or logistic are larger regression in case models are laboratory identifiable study. as The transparent scores points. laboratory Onlystudy a fewsupport appear to hyposis be non-significant, that differences se are only found factor inmethods factor method. For or bothmethods studies, are statistically factor methods significant, (b5, b6, with b7, b8) are discomfort ones that are prediction producing becoming worst discomfort worse as factor prediction. factor In addition, method is increased. differences methods are more pronounced for laboratory study than for field study. This is probably partly due to noise inherent to Table 4. Significant difference scores (%) each tested method (field study). field studies data. As observed in Tables b5 4 b6 5, b7 significant b8 b1000 difference b2000 scores b4000 dot3 not4 exceed t5 82%. This is due to fact that difference t two 36 eval 0 options 0 used 0 to0 derive 15 6 DGP will almost never be large significant, t5 11 since 0 this 16 index 40 was 13 designed 0 to be 0 robust 2 against 2 variations source detection methods. t4 Therefore, a score 79 80% 19 means 13 that for 7 all or 0 discomfort indices for each significance t3 test0 applied, 0 0 difference eval methods were significant. As already observedb4000 from 0 graphs, differences 0 are 2 larger in case laboratory study. The scores laboratory b study 2 support hyposis that differences factor methods or b1000 methods 0 0 are 7 statistically 27 significant, with discomfort prediction b becoming worse as factor factor method is increased. b7 2 6 From se results, it is clear that factor methods should be avoided. These methods provide b6 2 inappropriate source detections, which result in inaccurate discomfort predictions. Why se methods fail to accurately detect sources could be answered by following explanation. Table 5. Significant difference scores (%) each tested method (laboratory study). Table 4. Significant b5 difference b6 b7 scores b8 (%) b1000 b2000 eachb4000 tested method t3 t4 (field t5 study). t t5 b5 38 b6 60 b780 b80 b b2000 b t3 0 t4 t5 t6 t t5 t t4 b t b b b2000b b1000 b b8 b b7 2 6 b6 24 b6 2

13 Buildings 2018, 8, Table 5. Significant difference scores (%) each tested method (laboratory study). b5 b6 b7 b8 b1000 b2000 b4000 t3 t4 t5 t t t t b b Buildings 2018, 8, b1000 x FOR PEER 38REVIEW b b From se results, it is clear that factor methods should be avoided. These methods provide b6 24 inappropriate source detections, which result in inaccurate discomfort predictions. Why se methods fail to accurately detect sources could be answered by following explanation. For dim dim light light scenes, scenes, source sourcedetection detectionthreshold thresholdwhile while using using a factor a factor method method will will be be generally generally quite quite low, low, since since average average luminance luminance scene is scene low. is This low. will This ten will result tenin result an overdetection over-detection sources. sources. 10 shows 10an shows example an example a visual ascene visual in scene which in which computer computer screen in an screen was treated was treated as a as asource source by eval by eval factor 5 factor method, 5 method, although although visual scene visual was scene ranked was ranked as No discomfort as No discomfort by subject. by subject. 10. Glare source detection by eval factor method for visual scene field study. 10. Glare source detection by eval factor 5 method for a visual scene field study. On opposite, when a scene is bright, factor method will ten fail to find sources (under-detection). On opposite, Since when aaverage scene is luminance bright, factor method scene is will relatively ten fail high, to find sources (under-detection). detection threshold Since will be high average as well. luminance Eval will scene thus ismiss relatively some high, sources. The source visual detection scene threshold in will 11 was be high rated as well. as Disturbing, Eval willalthough thus missno some substantial sources. The sources visualwere scenedetected in by 11 was eva rated as Disturbing, factor 7 method. although no substantial sources were detected by eva factor 7 method. From results, it can also be inferred that choice an eval method can be done according to type discomfort that is most probable to occur in studied visual scene. It appears that task area method would be most appropriate for visual scenes, in which contrast is likely to occur. On or h, threshold method fers some advantages when being used for visual scenes with saturation, although task area method delivers results in a similar range. This observation was expected can be explained. In case contrast, i.e., in case low vertical illuminance scenes, like in field study, task area method would be more appropriate since adaptation level subject s eyes plays an important role in determination source luminance level at which is perceived. The task area method includes adaptation level in source detection process, by using a task area as reference for eyes adaptation level. 11. Glare source detection by eval factor 7 method for a visual scene laboratory study. From results, it can also be inferred that choice an eval method can be done

14 On opposite, when a scene is bright, factor method will ten fail to find sources (under-detection). Since average luminance scene is relatively high, source detection threshold will be high as well. Eval will thus miss some sources. The visual scene in Buildings 2018, 118, was 94 rated as Disturbing, although no substantial sources were detected 14 by 33 eva factor 7 method. Buildings 2018, 8, x FOR PEER REVIEW eyes plays an important role in determination source luminance level at which 11. Glare source detection by eval factor method for visual scene laboratory study. is perceived. 11. The Glare task source area detection method by includes eval factor adaptation 7 methodlevel for ain visual scene source laboratory detection study. process, by using From a task results, area as reference it can also for be inferred eyes adaptation that choice level. an eval method can be done On or h, for saturation, which occurs in scenes with relatively high vertical according to type discomfort that is most probable to occur in studied visual scene. illuminance, It appears that like in task laboratory area method study, would determination be most appropriate source for visual luminance scenes, threshold in which does contrast not directly is likely depend to occur. on On adaptation or h, subject s threshold eyes. Therefore, method fers an absolute some advantages threshold would when being fer better used results for visual for saturation scenes with saturation detection., Up to although now, most task common area method threshold delivers value has results been in 2000 a similar cd/m range. 2, 2, this is by This value observation is supported was expected by this study. can be explained. In case contrast, i.e., in case low vertical illuminance scenes, like in field 4.2. Background Luminance Definition study, task area method would be more appropriate since adaptation level subject s s compare two background luminance definitions by using boxplots for both field (left) laboratory (right) studies, three indicators accuracy discomfort prediction. More specifically, each boxplot is based on a group 44 datapoints, with each one being an indicator corresponding to one four discomfort indices (DGP is left out boxplots) one 11 eval methods (b5, b6, b7, b8, b1000, b2000, b4000, t3, t4, t5, t6). Non-significant statistics (Spearman correlation logistic regressions) are included in boxplots but are plotted as empty grey dots. The Theorange triangle points points represent represent indicators indicators based based on on DGP DGP index. index. At last, At last, lines lines connecting connecting two two adjacent adjacent boxplots boxplots represent evolution an anindicator when background luminance definition is varied. If If lines would cross each or, n influence background luminance definition would depend on eval method or on discomfort index used. Since lines are not crossing much in this case, influence background luminance definition is constant, whatever eval method or discomfort index is. 12. Boxplots Spearman rho for two background luminance definitions (field study). 12. Boxplots Spearman rho for two background luminance definitions (field study).

15 Buildings 2018, 8, Boxplots Spearman rho for two background luminance definitions (field study). 13. Boxplots Spearman rho for two background luminance definitions (laboratory study). REVIEW Spearman rho for two background luminance definitions (laboratory study). Buildings 2018,13. 8, xboxplots FOR PEER Buildings 2018, 8, x FOR PEER REVIEW Boxplots Boxplots AUC AUC for for two two background background luminance luminance definitions definitions (field (field study). study). 14. Boxplots AUC for two background luminance definitions (field study). 15. Boxplots AUC for two background luminance definitions (laboratory study). (laboratory study). study) Boxplots Boxplots AUC AUC for for two two background background luminance luminance definitions definitions (laboratory

16 Buildings 2018, 8, Boxplots AUC for two background luminance definitions (laboratory study). 16. Boxplots AICc for two background luminance definitions (field study). 16. Boxplots AICc for two background luminance definitions (field study). Buildings 2018, 8, x FOR PEER REVIEW Boxplots AICc for two background luminance definitions (laboratory study). 17. Boxplots AICc for two background luminance definitions (laboratory study). In each figure (s 12 17), overall accuracy discomfort prediction can be compared In each figure (s use 12 17), CIE overall definition accuracy background discomfort luminance prediction can its mamatical be compared definition. The use difference CIE definition two background background luminance luminance definitions its mamatical lies in fact definition. that The CIE difference definition takes into account two background cosine luminance correction definitions (Lambert s lies cosine in law) fact in that computation CIE definition takes background into account luminance. cosine correction But, it seems (Lambert s that this cosine difference law) in does computation not show in background accuracy luminance. discomfort But, prediction: it seems that this boxplots difference are similar does for not show field in laboratory accuracy studies discomfort for each prediction: statistical approach. boxplots On are anor similar note, for boxplots field laboratory laboratory studies study for are each more statistical spread than approach. those On anor field study. note, Therefore, boxplots variation laboratory in study accuracy are more discomfort spread than those prediction field study. Therefore, eval methods variation or in accuracy discomfort discomfort indices prediction are larger for laboratory eval study, methods as it or was already observed discomfort in s indices 4 9. are larger for laboratory study, as it was already observed in s Significance 4 9. tests looking at difference two background luminance definitions support Significance observations tests looking that are at made difference according to boxplots. two background The significant luminance difference definitions scores support (Tables 6 observations 7) do not exceed that are 13%, made according are mainly to 0%. This boxplots. illustrates The significant that difference difference scores (Tables two 6 definitions 7) do not is almost exceed never 13%, significant, are mainly depending 0%. This on illustrates which significance that difference test is applied. This two is confirmed definitions by is looking almost never more significant, closely to depending boxplots on that which are significance related to test AICc is applied. indicator: This is confirmed variation in by looking AICc more values closely to boxplots use that CIE are definition related to or AICc mamatical indicator: definition variation is in never AICc larger values than 10 points. use CIE definition or mamatical definition is never larger than 10 points. Table 6. Significant difference scores (%) two background luminance definitions (field study). b5 b6 b7 b8 b1000 b2000 b4000 t3 t4 t5 t6 CIE Lb ><

17 Buildings 2018, 8, Table 6. Significant difference scores (%) two background luminance definitions (field study). CIE Lb >< Math. Lb b5 b6 b7 b8 b1000 b2000 b4000 t3 t4 t5 t Table 7. Significant difference scores (%) two background luminance definitions (lab. study). b5 b6 b7 b8 b1000 b2000 b4000 t3 t4 t5 t6 CIE Lb >< Math. Lb Buildings 2018, 8, x FOR PEER REVIEW luminance From sedistribution observations, it appears background that re a is visual no significant scene is extremely difference heterogeneous, in accuracyfor discomfort instance, when background prediction luminance values use are much CIEhigher but definition is still not mamatical considered definition as a source in background one specific luminance. part Both field could view. be used alternatively without observing any change in results a study. However, influence background luminance definition, namely influence 4.3. Search Radius use cosine correction in background luminance computation, should be checked in case extreme visual scenes. Indeed, this parameter could have an influence when s show boxplots that are built on same principals than those in s 12 17, luminance distribution background a visual scene is extremely heterogeneous, for instance, but in this case, eval parameter being varied is search radius. In each figure (s 18 when background luminance values are much higher but is still not considered as a source in 23), overall accuracy discomfort prediction can be compared use a search one specific part field view. radius having an opening angle 0.06, 0.2, or 0.3 radians. By varying search radius, maximal 4.3. distance Search within Radius which two nearby glaring pixels are merged in one source is varied. Therefore, smaller search radius, smaller solid angle detected sources but more m. s show boxplots that are built on same principals than those in s 12 17, but in The this default case, search eval radius parameter in eval being is varied 0.2 radians, is search it radius. appears In from each figure boxplots (s that 18 23), this default overall parameter accuracy is appropriate. discomfort It is prediction one can providing be compared best accuracy use discomfort a search radius having prediction an opening in field angle 0.06, laboratory 0.2, or studies. 0.3 radians. However, By varying several lines search are radius, crossing, maximal especially distance in within laboratory which study, two nearby as some glaring lines go pixels up while are merged ors in go one down when source varying is varied. Therefore, search radius. smaller This is a sign search that radius, influence smaller search solid radius angle is not constant. detected sources but more m. 18. Boxplots Spearman rho for three search radius (field study). 18. Boxplots Spearman rho for three search radius (field study).

18 Buildings 2018, 8, Boxplots Spearman rho for three search radius (field study). 19. Boxplots Spearman rho for three search radius (laboratory study). 19. Boxplots Spearman rho for three search radius (laboratory study). Buildings 2018, 8, x FOR PEER REVIEW Buildings 2018, 8, x FOR PEER REVIEW 20. Boxplots AUC for three search radius (field study). 20. Boxplots AUC for three search radius (field study). 20. Boxplots AUC for three search radius (field study). 21. Boxplots AUC for three search radius (laboratory study). 21. Boxplots AUC for three search radius (laboratory study). 21. Boxplots AUC for three search radius (laboratory study)

19 Buildings 2018, 8, Boxplots AUC for three search radius (laboratory study). 22. Boxplots AICc for three search radius (field study). 22. Boxplots AICc for three search radius (field study). Buildings 2018, 8, x FOR PEER REVIEW Boxplots AICc for three search radius (laboratory study). 23. Boxplots AICc for three search radius (laboratory study). Looking at significant difference scores three search radius parameters (Tables 8 The default 9), it appears searchthat radius this parameter in eval does is 0.2 not radians, produce significant it appears differences from on boxplots accuracy that this default discomfort parameter isprediction, appropriate. at It least is not one for providing factor best threshold accuracy methods. discomfort When using prediction task in area field method, laboratory differences studies. might However, be interpreted several as statistically lines aresignificant crossing, especially in some cases. in These laboratory cases study, correspond as someto lines lines go upwith while ors steepest goslopes downin when boxplot varying graphs. search radius. This is a sign that influence search radius is not constant. Looking at Table 8. significant Significant difference scores scores (%) three three search search radius radius (field parameters study). (Tables 8 9), it appears that this parameter b5 b6 b7 does b8 not b1000 produce b2000 significant b4000 differences t3 t4 t5 ont6 accuracy discomfort r0.2 prediction, >< r0.06 at 0 least 12 not 7 for0 factor 0 2 threshold 10 methods. 2 2 When 2 2 using task area method, r0.2 differences >< r0.3 might 2 0 be interpreted 0 0 as 0 statistically 0 significant 0 62 in 22 some 0 cases. 0 These cases correspond to r0.06 lines >< r0.3 with 0 steepest 13 7 slopes 11 in0 boxplot 0 graphs Table Significant difference scores (%) (%) three three search search radius radius (laboratory (field study). b5 b6 b7 b8 b1000 b2000 b4000 t3 t4 t5 t6 b5 b6 b7 b8 b1000 b2000 b4000 t3 t4 t5 t6 r0.2 >< r r0.2 r0.2 >< r0.06 >< r r0.2 r0.06 >< r0.3 >< r r0.06 >< r Although differences in accuracy prediction could not be demonstrated as being significant in a majority cases, re seems to be a tendency towards a better accuracy discomfort prediction when using default search radius. However, using only three variations search radius is not enough to grasp full extent influence search radius on source detection, indirectly on accuracy discomfort prediction. Moreover,

20 Buildings 2018, 8, Table 9. Significant difference scores (%) three search radius (laboratory study). b5 b6 b7 b8 b1000 b2000 b4000 t3 t4 t5 t6 r0.2 >< r r0.2 >< r r0.06 >< r Although differences in accuracy prediction could not be demonstrated as being significant in a majority cases, re seems to be a tendency towards a better accuracy discomfort prediction when using default search radius. However, using only three variations search radius is not enough to grasp full extent influence search radius on source detection, indirectly on accuracy discomfort prediction. Moreover, influence search radius most probably depends on visual scene, especially on distance two contrasted pixels in a luminance map. In case this study, observers from both datasets were sitting close to façade, thus at a relatively close distance from blinds. But, this might not be case for large open-plan fices. If observer is located furr away from blinds, distance contrasted pixels blinds will vary in luminance map, opening angle search radius will have to be varied in order to produce a similar result than if observer was sitting close to façade. An extended study on appropriate search radius for source detection would be required to thoroughly underst influence this parameter. In meantime, it seems reasonable to keep using default eval search radius having an opening angle 0.2 radians Task Area Size Buildings 2018, 8, x FOR PEER REVIEW s show boxplots that are similar than ones in s 12 23, for both field (left) 4.4. Task Area Size laboratory (right) studies, for three indicators accuracy discomfort prediction. However, in s case24 29 show taskboxplots area size that parameter, are similar than each boxplot ones in is s based12 23, on afor group both 16 field datapoints. (left) laboratory (right) studies, for three indicators accuracy discomfort The 16 datapoints are indicators corresponding to four discomfort indices (DGI, CGI, prediction. However, in case task area size parameter, each boxplot is based on a group DGImod, 16 datapoints. UGP) The 16 datapoints four taskare area methods indicators corresponding (t3, t4, t5, t6). to The four factor discomfort threshold indices methods are left out (DGI, boxplots CGI, DGImod, since UGP) variation four duetask to area task methods area(t3, size t4, t5, has t6). no The influence factor on threshold ir accuracy discomfort methods prediction. left out As boxplots few since non-significant variation due statistics to task (Spearman area size has correlation no influence on logistic ir accuracy discomfort prediction. As few non-significant statistics (Spearman regressions) are found when using factor method, all datapoints plotted in s are correlation logistic regressions) are found when using factor method, all datapoints based onplotted significant in s statistics are At based last, on significant orangestatistics. triangleat points last, represent orange triangle indicators points represent based on DGP index. indicators based on DGP index. 24. Boxplots Spearman rho for three task area sizes (field study). 24. Boxplots Spearman rho for three task area sizes (field study).

21 Buildings 2018, 8, Boxplots Spearman rho for three task area sizes (field study). 25. Boxplots Spearman rho for three task area sizes (laboratory study). 25. Boxplots Spearman rho for three task area sizes (laboratory study). Buildings 2018, 8, x FOR PEER REVIEW Buildings 2018, 8, x FOR PEER REVIEW 26. Boxplots AUC for three task area sizes (field study). 26. Boxplots AUC for three task area sizes (field study). 26. Boxplots AUC for three task area sizes (field study). 27. Boxplots AUC for three task area sizes (laboratory study). 27. Boxplots AUC for three task area sizes (laboratory study). 27. Boxplots AUC for three task area sizes (laboratory study)

22 Buildings 2018, 8, Boxplots AUC for three task area sizes (laboratory study). 28. Boxplots AICc for three task area sizes (field study). 28. Boxplots AICc for three task area sizes (field study). Buildings 2018, 8, x FOR PEER REVIEW Boxplots AICc for three task area sizes (laboratory study). 29. Boxplots AICc for three task area sizes (laboratory study). The overall accuracy discomfort prediction can be compared on boxplots The three overall different accuracy task area sizes, discomfort having, respectively, prediction an opening can be compared angle 30 on (0.52 rad.), boxplots 60 (1 rad.), three 90 different (1.57 rad.). taskthe area60 sizes, task having, area opening respectively, angle an is what opening is believed angle to 30be (0.52 commonly rad.), 60used (1 rad.), in eval, 90 (1.57 as rad.). this corresponds The 60 task to area ergorama. opening angle Varying is what task is believed area size might to be commonly be interpreted used as in eval, varying as this size corresponds part to field ergorama. view to which Varying subject s task eyes areaare size adapting. might be But, interpreted increasing as varying this task area size size does part not systematically field view mean to which increasing subject s average eyesluminance are adapting. this But, task increasing area; this task visual area scene sizewill does be not determinant systematically in this mean case. s increasing average help to underst luminance this implication task area; visual varying scene will task be area determinant size on inaverage this case. task s luminance, 30 31thus helpon to underst source detection implication threshold. varying For task area laboratory size onstudy average ( task 31), luminance, variation thus task onarea size has source a small detection threshold. relatively For constant laboratory impact on study task ( luminance: 31), variation larger task task area area size, size larger has a small task relatively luminance. constant In impact case on field task study luminance: ( 30), larger impact task variation area size, task larger area size task luminance. on task In luminance case is larger, field but study more ( chaotic. 30), This is impact due to fact variation that, in real fice task settings, area size field view changes a lot from one desk to or, hence luminance distribution this on task luminance is larger, but more chaotic. This is due to fact that, in real fice settings, field view varies a lot as well. In a laboratory setting, on contrary, visual field is relatively field view changes a lot from one desk to or, hence luminance distribution this constant, causing variation in task luminance to be relatively constant as well. field view varies a lot as well. In a laboratory setting, on contrary, visual field is relatively constant, causing variation in task luminance to be relatively constant as well.

23 threshold. For laboratory study ( 31), variation task area size has a small relatively constant impact on task luminance: larger task area size, larger task luminance. In case field study ( 30), impact variation task area size on task luminance is larger, but more chaotic. This is due to fact that, in real fice settings, field view changes a lot from one desk to or, hence luminance distribution this Buildings field 2018, view 8, 94 varies a lot as well. In a laboratory setting, on contrary, visual field is relatively constant, causing variation in task luminance to be relatively constant as well. 30. Comparison task area average luminance three different task area sizes 30. Comparison task area average luminance three different task area sizes for all evaluations field study. for all evaluations field study. Buildings 2018, 8, x FOR PEER REVIEW Comparison task area average luminance 3 different task area sizes for 31. Comparison task area average luminance 3 different task area sizes for all evaluations lab study. all evaluations lab study. However, it seems that a consistent variation source detection threshold due to a However, consistent variation it seems that task a consistent luminance does variationot produce a consistent source detection influence on threshold due accuracy to a consistent discomfort variation prediction. task luminance does In boxplots not produce laboratory a study consistent (s influence 25, 27 on 29), accuracy discomfort lines are crossing prediction. going In in opposite boxplots directions, laboratory which means study that (s influence 25, 27 task 29), area lines size is not consistent throughout discomfort indices or task area methods. Moreover, are crossing going in opposite directions, which means that influence task area size is three indicators accuracy discomfort prediction disagree on which would be best not consistent throughout discomfort indices or task area methods. Moreover, three task area size. At last, significant difference scores three task area sizes for indicators laboratory study accuracy (Table 10) discomfort are low or equal prediction to 0%. No conclusions disagree oncan which refore would be be drawn best from task area size. At analysis last, significant task area size difference influence scores while using laboratory three task dataset. area sizes for laboratory study (Table 10) On are lowor or equal h, trom 0%. Noboxplots conclusions can field refore study (s be drawn 24, 26 from 28), analysis a steady trend task area size can be influence observed: while by decreasing using laboratory task area dataset. size, accuracy discomfort prediction is increased. It is however quite difficult to explain this trend. What could be observed in 30 is that Table when 10. Significant task luminance difference is relatively scores (%) low, using a smaller three task area sizes (laboratory tends increase study). this task luminance; but when task luminance is relatively high, using a smaller task area size tends to decrease task luminance. A small task t3area size t4 thus t5 leads to t6a more constant task luminance throughout different visual scenes. 60 X 30 A more accurate 0 6 discomfort 6 0 prediction could refore be achieved thanks to use 60 a less X 90 fluctuating 2 eye adaptation 0 24 level. 13In reality, it is indeed expected that subject s eyes adapt to 30 Xcomputer 90 screen 0 brightness, 0 0 which 0should not vary a lot. However, differences three task area sizes do not appear to be statistically significant, even for field study (Table 11). More insight on actual task area, namely area field view to which subject s eyes are really adapting, are required to better underst what can be seen here, to determine an appropriate task area size. Table 10. Significant difference scores (%) three task area sizes (laboratory study).

24 Buildings 2018, 8, On or h, from boxplots field study (s 24, 26 28), a steady trend can be observed: by decreasing task area size, accuracy discomfort prediction is increased. It is however quite difficult to explain this trend. What could be observed in 30 is that when task luminance is relatively low, using a smaller task area size tends to increase this task luminance; but when task luminance is relatively high, using a smaller task area size tends to decrease task luminance. A small task area size thus leads to a more constant task luminance throughout different visual scenes. A more accurate discomfort prediction could refore be achieved thanks to use a less fluctuating eye adaptation level. In reality, it is indeed expected that subject s eyes adapt to computer screen brightness, which should not vary a lot. However, differences three task area sizes do not appear to be statistically significant, even for field study (Table 11). More insight on actual task area, namely area field view to which subject s eyes are really adapting, are required to better underst what can be seen here, to determine an appropriate task area size. Table 11. Significant difference scores (%) three task area sizes (field study). t3 t4 t5 t6 60 X X Buildings 2018, 8, x FOR PEER REVIEW 30 X Smooth Option 4.5. Smooth Option s compare accuracy discomfort prediction when detected sources s are smood compare or not. Using accuracy smooth discomfort parameter prediction causes when pixels that detected are not detected sources as are glaring smood pixels by or not. algorithm, Using smooth but that parameter are nested causes inside an area pixels that glaring are pixels not detected to be considered as glaring pixels as glaring by pixels algorithm, as well but that included are nested in inside surrounding an area glaring source. pixels Therefore, to be considered when assmooth glaring pixels parameter as well is applied, included a detected in surrounding source in a luminance source. Therefore, map will have whena larger smooth solid parameter angle, but is a lower applied, average a detected luminance. The source smooth in a parameter luminanceis map presumably will have seldom a larger used, solid mainly angle, because but a lower it did average not belong luminance. to recommended The smooth options parameter in eval, is presumably little seldom data used, have mainly been published because itabout did not it, belong making toits contribution recommended not well options understood. in eval, In addition, little its data computation have been published time three about to four it, making times its longer. contribution not well understood. In addition, its computation time is three to four times longer. 32. Boxplots Spearman rho for non-smooth/smooth parameter (field study). 32. Boxplots Spearman rho for non-smooth/smooth parameter (field study).

25 Buildings 2018, 8, Boxplots Spearman rho for non-smooth/smooth parameter (field study). 33. Boxplots Spearman rho for non-smooth/smooth parameter (laboratory study). 33. Boxplots Spearman rho for non-smooth/smooth parameter (laboratory study). Buildings 2018, 8, x FOR PEER REVIEW Buildings 2018, 8, x FOR PEER REVIEW Boxplots AUC for non-smooth/smooth parameter (field study). 34. Boxplots AUC forfor parameter study). 34. Boxplots AUC non-smooth/smooth non-smooth/smooth parameter (field(field study). parameter(laboratory (laboratorystudy). study) Boxplots Boxplots AUC AUC for for non-smooth/smooth non-smooth/smooth parameter 35. Boxplots AUC for non-smooth/smooth parameter (laboratory study).

26 Buildings 2018, 8, Boxplots AUC for non-smooth/smooth parameter (laboratory study). 36. Boxplots AICc for non-smooth/smooth parameter (field study). 36. Boxplots AICc for non-smooth/smooth parameter (field study). Buildings 2018, 8, x FOR PEER REVIEW Boxplots AICc for non-smooth/smooth parameter (laboratory study). 37. Boxplots AICc for non-smooth/smooth parameter (laboratory study). From boxplots (s 32 37), a general trend is observed towards a higher accuracy discomfort From boxplots prediction (s with 32 37), smooth a general parameter. trendthis observed is especially towards visible a higher for accuracy field study, as discomfort lines all go prediction in same with direction. smooth This parameter. implies that is influence especially visible smooth for option field study, is not dependent as lineson all godiscomfort in same direction. index that Thisis implies used or that eval influence method applied. smoothto option underst is not why dependent smooth on parameter discomfort fers a index constant thatimprovement used or eval discomfort method applied. prediction, underst its impact on why four smooth main parameter physical quantities fers a constant influencing improvement discomfort discomfort [1] was investigated prediction, its through impact on Glare four Impact main physical (GI). The quantities GI was first influencing introduced discomfort by [32], [1] it was was investigated redefined here, through according Glare to Equation Impact (GI). (2). The GI was first introduced by [32], it was redefined here, according to Equation (2). L2 s,i ω i GI = L2 s,i ω i LL b P (2) b P 2 (2) i i with Ls,i, luminance source; with L s,i, luminance source; ωi, solid angle source; Lb, luminance background;, Pi, position index related to source. First, variation GI according to smooth parameter was studied. This did not help understing influence smooth parameter, since variations GI due to smoothing were not constant very dependent on visual scene. In bright visual scenes, a

27 Buildings 2018, 8, ω i, solid angle source; L b, luminance background;, P i, position index related to source. First, variation GI according to smooth parameter was studied. This did not help understing influence smooth parameter, since variations GI due to smoothing were not constant very dependent on visual scene. In bright visual scenes, a squared decrease in average source luminance due to use smooth parameter was generally balanced by increase in solid angle this source. However, in low-light scenes, squared decrease in source luminance could most time not be entirely balanced by increase in solid angle. The only constant variation that is brought by smooth parameter is decrease in background luminance, as can be seen in s Since originally non-glaring pixels located next to a source are included in this source due to smoothing, average background luminance is lowered as some brightest pixels Buildings 2018, 8, x FOR PEER REVIEW background those located next to a source are retrieved from this background. The constant decrease despite a inclear background tendency towards luminance a better due to prediction smoothingwith could thus smooth be linked parameter to constant applied increase (s in 32 37). accuracy discomfort perception. 38. Comparison background luminance while varying smooth parameter with 38. Comparison background luminance while varying smooth parameter with t4 t4 method (task area method with a factor 4), for all evaluations field study. method (task area method with a factor 4), for all evaluations field study. Looking at significant difference scores smooth/non-smooth parameter (Tables 12 13), differences in accuracy discomfort prediction can be considered to be statistically significant only for a few cases, especially in field study. The smooth parameter appears not to produce a strong significant difference in accuracy discomfort prediction, despite a clear tendency towards a better prediction with smooth parameter applied (s 32 37). Table 12. Significant difference scores (%) smooth/non-smooth parameter (field study). Smooth >< Non-smooth b5 b6 b7 b8 b1000 b2000 b4000 t3 t4 t5 t

28 Buildings 2018, 38. 8, 94 Comparison background luminance while varying smooth parameter with t4 method (task area method with a factor 4), for all evaluations field study. 39. Comparison background luminance while varying smooth parameter with 39. Comparison background luminance while varying smooth parameter with b2000 method (method with threshold 2000 cd/m ), for all evaluations laboratory study. b2000 method (method with a threshold 2000 cd/m 2 ), for all evaluations laboratory study. Table 12. Significant difference scores (%) smooth/non-smooth parameter (field study). Table 13. Significant difference scores (%) smooth/non-smooth parameter (laboratory study). b5 b6 b7 b8 b1000 b2000 b4000 t3 t4 t5 t6 Smooth b5 b6 b7 b8 b1000 b2000 b4000 t3 t4 t5 t6 Smooth >< Non-smooth >< Non-smooth The observations that are made here following investigation smooth parameter should not lead to an excessive indiscriminate use this parameter, with sole purpose to improve correlation subjective ratings discomfort indices in a study. These results should first be validated in an extended study. Moreover, settings choice in eval should always be done in an objective sensible way, so as not to bias results. At last, results that are presented here should be interpreted with respect to limitations study. This study concentrated on most common settings that are available in eval. However, or settings such as definition position index might also influence accuracy discomfort prediction. Moreover, observations are made based on a limited number visual scenes. It is believed that use two different datasets brings a larger diversity in studied visual scenes ir luminance distributions, making results applicable to a large number situations. But, different or extreme visual scenes could produce different results. Finally, se results are based on five most common discomfort indices for daylighting. However, se indices are not performing as well as could be expected [33], a lot research is still going on to underst mechanism behind discomfort perception to determine an accurate way measuring it. Moreover, one se five indices, DGP, possesses a validity range, which means that DGP values lower than 0.2 or higher than 0.8 should be interpreted with an additional layer caution. 5. Conclusions In this study, influence choice eval methods parameters on accuracy discomfort prediction was investigated. The aim was to determine which method parameters fer best discomfort prediction for both types discomfort, namely contrast

29 Buildings 2018, 8, saturation. Therefore, two datasets were used in this study, each one representative one type. In total, 63 different eval options were tested (Table 2), for each option, five common daylight discomfort indices were calculated (DGP, DGI, CGI, DGImod, UGP). Three statistical indicators were computed for each index, for each eval option, for both datasets, so that accuracy discomfort prediction deriving from each eval option can be evaluated. These three statistical indicators are Spearman correlation coefficient (or Spearman rho), Area Under ROC Curve a binomial logistic regression model (or AUC), corrected Akaike s Information Criterion an ordinal logistic regression model (or AICc). The results confirmed that task area method should be preferred when evaluating visual scenes with contrast. For visual scenes where saturation is more predominant, threshold method seems most appropriate one, although task area method fers very similar results. More specifically, results showed that task area method should be applied with a multiplying factor 4 or 5, while most appropriate threshold for threshold method seems to be 2000 cd/m 2, as is generally found in literature. The calculation method background luminance CIE definition or mamatical definition seems to have no influence on accuracy discomfort prediction. A small trend could be observed for influence search radius task area size parameter, although ir influence on accuracy discomfort prediction could not be demonstrated as being statistically significant. The default search radius in eval, having an opening angle 0.2 radians, was one leading to best discomfort prediction. As for task area size, it seems that a smaller task area size, having an opening angle 30 60, would be more appropriate for discomfort prediction. However, more research involving datasets with different visual conditions is required to confirm se results. At last quite surprisingly, applying smooth parameter appeared to generate a more accurate discomfort prediction. It was hyposized that this effect might be due to decrease in background luminance when applying smoothing. Rar than resulting in an unreasonable use smooth parameter to improve discomfort studies results, this observation should initiate reconsideration basis most discomfort indices nowadays, namely Glare Impact formula (Equation (2)). Since it was observed that, for both studies, accuracy discomfort prediction was most certainly improved by decreasing background luminance, weighting or even use this physical quantity in discomfort models could be questioned. An extended laboratory study reassessing influence each one four main variables that is used in discomfort models (Ls, ω, Lb, P) would be required, to validate basis discomfort knowledge to provide a clear definition a source. In addition, authors would like to emphasize importance choice eval methods parameters. When using eval, one should always be aware that this tool could lead to very different results in discomfort index calculation depending on se choices. s show extreme examples such variations in discomfort indices values that were observed in framework this study. Although DGP was found to be only slightly influenced by detection parameters methods in eval, settings could matter as well for specific cases. In 40, an extreme visual scene from each dataset would change from disturbing category to intolerable category (thresholds defined according to [34]) due to use different settings in eval. More extreme variations were observed for four or discomfort indices (DGI, CGI, DGImod, UGP) ( 41). In given examples (s 42 43), no source was found by using b1000_lb method for field study example ( 42, left) or b7_lb method for laboratory study example ( 43, right). The value CGI index corresponding to se two scenes is 0, y are refore categorized as Imperceptible ( 41). But, for or settings, such as t3_smood method for field study example ( 42, right) b1000_r0.06 method for laboratory study example ( 43, left), sources are detected in scenes. The same scenes are n categorized as Unacceptable or even Uncomfortable (thresholds defined according to [17,35]) ( 41). It is refore

30 42 43), no source was found by using b1000_lb method for field study example ( 42, left) or b7_lb method for laboratory study example ( 43, right). The value CGI index corresponding to se two scenes is 0, y are refore categorized as Imperceptible ( 41). But, for or settings, such as t3_smood method for field study example ( 42, right) b1000_r0.06 method for laboratory study example Buildings 2018, 8, ( 43, left), sources are detected in scenes. The same scenes are n categorized as Unacceptable or even Uncomfortable (thresholds defined according to [17,35]) ( 41). It is refore strongly recommended to create a check file when using eval to strongly recommended to create a check file when using eval to calculate discomfort indices calculate discomfort indices from a luminance map examine visually source(s) fromdetected. a luminance map examine visually source(s) detected. The visual analysis The visual analysis sources is a useful way to verify that sources detected sources are is areasonable. useful way to verify that sources detected are reasonable. DGP 40. DGP same visualscene scenefor for different different eval parameters (field(field same visual evalmethods methods parameters Buildings 2018,40. 8, x FOR PEER REVIEW laboratory studies). laboratory studies). Buildings 2018, 8, x FOR PEER REVIEW Commission Illumination Glare Glare Index same visual scene for different 41. Commission ononillumination Index(CGI) (CGI) same visual scene for different 41. Commission Illumination Index (CGI) same visual scene for different eval methods on parameters (field Glare laboratory studies). eval methods parameters (field laboratory studies). eval methods parameters (field laboratory studies). 42. Extreme example (field study) difference in source(s) detection resulting from b1000_lb setting (left) t3_smood setting (right) in eval. 42. Extreme example (field study) difference in source(s) detection resulting from 42. Extreme example (field study) difference in source(s) detection resulting from b1000_lb b1000_lb setting setting (left) (left) t3_smood t3_smood setting setting (right) (right) in in eval. eval.

31 Buildings 2018, 42. 8, 94 Extreme example (field study) difference in source(s) detection resulting from b1000_lb setting (left) t3_smood setting (right) in eval. 43. Extreme example (laboratory study) difference in source(s) detection resulting 43. Extreme example (laboratory study) difference in source(s) detection resulting from b1000_r0.06 setting (left) b7_lb setting (right) in eval. from b1000_r0.06 setting (left) b7_lb setting (right) in eval. To conclude, for predominating saturation scenes, it would be beneficial to use an absolute threshold To conclude, 2000 cd/m for predominating 2. This method saturation should also be preferred scenes, it would for simulation be beneficial work to with use an an observer absolute threshold location close 2000 to cd/m façade. 2. This Using method should task area also method be preferred with for a factor simulation 4 or 5 work delivers with for an all observer location investigated close cases to (saturation façade. Using contrast task area method scenes) with reasonable a factor results 4 or 5 delivers it is for preferred all investigated option for predominating cases (saturation contrast contrast scenes. This scenes) method reasonable should be results applied in it simulation is preferred work, option when for observer predominating is located contrast furr away scenes. from This façade, method although should be applied task area in simulation definition might work, when be sensitive. observer Using is located factor furr method eval away from default façade, method was although found task area to lead definition to significantly might be sensitive. lower performances Using factor method eval is refore not recommended. default method was From found this study, to lead to significantly influences lower performances background luminance is refore definition, not recommended. search radius, From this study, task area influences size were found background to be not luminance statistically definition, significant. It is search refore radius, recommended task to use area size default were setting found to for be se not parameters. statistically significant. Furr research It is on refore definition recommended a to source use in daylighting default setting is required, for se so that parameters. se parameters Furr research can be determined on definition knowingly a reliably. source in daylighting is required, so that se parameters can be determined knowingly reliably. Author Contributions: Conceptualization, C.P., J.W. M.B.; Methodology, C.P., J.W. M.B.; Formal Analysis, C.P.; Investigation, C.P. J.W.; Data Curation, C.P. J.W.; Writing-Original Draft Preparation, C.P.; Writing-Review & Editing, J.M. M.B.; Visualization, C.P.; Supervision, J.W. M.B.; Project Administration, C.P.; Funding Acquisition, C.P. M.B. Funding: This research was funded by Fonds de la Recherche Scientifique FNRS. Acknowledgments: The DE-Quanta study (laboratory study) was supported by Fraunher-Institute for Solar Energy Systems ISE Deutsche Forschungsgemeinschaft DFG (WA1155/5-1). Conflicts Interest: The authors declare no conflict interest. The founding sponsors had no role in design study; in collection, analyses, or interpretation data; in writing manuscript, in decision to publish results. References 1. Commission Internationale de l Eclairage (CIE). Discomfort Glare in Interior Working Environment; Commission Internationale de l Eclairage: Vienna, Austria, 1983; p Pierson, C.; Wienold, J.; Bodart, M. Review factors influencing perceived discomfort from daylighting. J. Illum. Eng. Soc. N. Am. 2018, 14, Wienold, J.; Christfersen, J. Evaluation methods development a new prediction model for daylight environments with use ccd cameras. Energy Build. 2006, 38, [CrossRef] 4. Wienold, J.; Reetz, C.; Kuhn, T.E.; Christfersen, J. Eval A new radiance-based tool to evaluate daylight in fice spaces. In Proceedings 3rd International Radiance Workshop, Fribourg, Switzerl, October Wienold, J. New features eval. In Proceedings 11th International Radiance Workshop, Copenhagen, Denmark, September Wienold, J.; Andersen, M. Eval 2.0 new features faster more robust hdr-image evaluation. In Proceedings 15th International Radiance Workshop, Padua, Italy, August Wienold, J. Daylight Glare in Offices. Ph.D. Thesis, Universität Karlsruhe (TH), Freiburg, Germany, 2009.

32 Buildings 2018, 8, Jakubiec, J.A.; Reinhart, C.F. Diva 2.0: Integrating daylight rmal simulations using rhinoceros 3d energyplus. In Proceedings Building Simulation th Conference International Building Performance Simulation Association, Sydney, Australia, November Pierson, C.; Piderit, M.B.; Wienold, J.; Bodart, M. Discomfort from daylighting: Influence culture on discomfort perception. In Proceedings CIE 2017 Midterm Meeting, Jeju, Korea, October Moosmann, C.; Wienold, J.; Wagner, A.; Wittwer, V. AbschluErmittlung Relevanter Einflussgrößen auf Die Subjektive Bewertung von Tageslicht zur Bewertung des Visuellen Komforts in Büroräumen. Abschlussbericht Available online: (accessed on 6 June 2018). 11. Dubois, M.-C. Impact Shading Devices on Daylight Quality in Office Simulations with Radiance; TABK 01/3062; Lund University: Lund, Sweden, 2001; p Shin, J.Y.; Yun, G.Y.; Kim, J.T. View types luminance effects on discomfort assessment from windows. Energy Build. 2012, 46, [CrossRef] 13. Bülow-Hübe, H. Daylight in Glazed Office Buildings A Comparative Study Daylight Availability, Luminance Illuminance Distribution for an Office Room with Three Different Glass Areas; EBD-R 08/17; Lund University: Lund, Sweden, 2008; p Lee, E.S.; Clear, R.D.; Fernes, L.; Ward, G. Commissioning Verification Procedures for Automated Roller Shade System at New York Times Headquarters, New York; Lawrence Berkeley National Laboratory (LBNL): Berkeley, CA, USA, 2007; p Van Den Wymelenberg, K.; Inanici, M.; Johnson, P. The effect luminance distribution patterns on occupant preference in a daylit fice environment. J. Illum. Eng. Soc. 2010, 7, Chauvel, P.; Collins, J.B.; Dogniaux, R.; Longmore, J. Glare from windows: Current views problem. Light. Res. Technol. 1982, 14, [CrossRef] 17. Einhorn, H.D. Discomfort : A formula to bridge differences. Light. Res. Technol. 1979, 11, [CrossRef] 18. Fisekis, K.; Davies, M.; Kolokotroni, M.; Langford, P. Prediction discomfort from windows. Light. Res. Technol. 2003, 35, [CrossRef] 19. Hirning, M.B.; Isoardi, G.L.; Cowling, I. Discomfort in open plan green buildings. Energy Build. 2014, 70, [CrossRef] 20. Siegel, S. Nonparametric Statistics for Behavioral Sciences; McGraw-Hill Book Company: New York, NY, USA, Ferguson, C.J. An effect size primer: A guide for clinicians researchers. Pr. Psychol. Res. Pract. 2009, 40, [CrossRef] 22. Diedenhen, B.; Musch, J. Cocor: A comprehensive solution for statistical comparison correlations. PLoS ONE 2015, 10, e [CrossRef] [PubMed] 23. Hosmer, D.W.; Lemeshow, S. Applied Logistic Regression, 2nd ed; Wiley: Toronto, ON, Canada, Safari, S.; Baratloo, A.; Elfil, M.; Negida, A. Evidence based emergency medicine; part 5 receiver operating curve area under curve. Emergency 2016, 4, [PubMed] 25. Venkatraman, E.S.; Begg, C.B. A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment. Biometrika 1996, 83, [CrossRef] 26. DeLong, E.R.; DeLong, D.M.; Clarke-Pearson, D.L. Comparing areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics 1988, 44, [CrossRef] [PubMed] 27. Pepe, M.; Longton, G.; Janes, H. Estimation Comparison Receiver Operating Characteristic Curves; Biostatistics Working Paper Series; The Berkeley Electronic Press: Berkeley, CA, USA, Murtaugh, P.A. In defense p values. Ecology 2014, 95, [CrossRef] [PubMed] 29. Burnham, K.P.; Anderson, D.R. Model Selection Multimodel Inference: A Practical Information-Theoretic Approach, 2nd ed.; Springer-Verlag: New York, NY, USA, Brewer, M.J.; Butler, A. Model Selection Cult Aic; Universitat Autònoma de Barcelona: Barcelona, Spain, Hurvich, C.M.; Tsai, C.-L. Regression time series model selection in small samples. Biometrika 1989, 76, [CrossRef]

33 Buildings 2018, 8, Sarey Khanie, M.; Jia, Y.; Wienold, J.; Andersen, M. A sensitivity analysis on detection parameters. In Proceedings BS th International Conference International Building Performance Simulation Association, Hyderabad, India, 7 9 December Suk, J.Y.; Schiler, M.; Kensek, K. Investigation existing discomfort indices using human subject study data. Build. Environ. 2017, 113, [CrossRef] 34. Reinhart, C.; Wienold, J. The daylighting dashboard A simulation-based design analysis for daylit spaces. Build. Environ. 2011, 46, [CrossRef] 35. Boyce, P.R. Human Factors in Lighting, 3rd ed.; Taylor & Francis Group: Boca Raton, FL, USA, 2014; p by authors. Licensee MDPI, Basel, Switzerl. This article is an open access article distributed under terms conditions Creative Commons Attribution (CC BY) license (