On Quantifying the Accuracy of Maximum Likelihood Estimation of Participant Reliability in Social Sensing

Transcription

1 On Quantfyng the Accuracy of Maxmum Lkelhood Estmaton of Partcpant Relablty n Socal Sensng Dong Wang, Tarek Abdelzaher Department of Computer Scence Unversty of Illnos Urbana, IL dwang24@llnos.edu, zaher@llnos.edu Lance Kaplan Charu C. Aggarwal Networked Sensng & Fuson IBM Research Branch Yorktown Heghts, NY US Army Research Laboratory charu@us.bm.com Adelph, MD lance.m.kaplan@us.army.ml ABSTRACT Ths paper presents a confdence nterval quantfcaton of maxmum lkelhood estmaton of partcpant relablty n socal sensng applcatons. The work s motvated by the emergence of socal sensng as a data collecton paradgm, where humans perform the data collecton tasks. A key challenge n socal sensng applcatons les n the uncertan nature of human measurements. Unlke well-calbrated and well-tested nfrastructure sensors, humans are less relable, and the lkelhood that partcpants measurements are correct s often unknown a pror. Hence, t s hard to estmate the accuracy of conclusons made based on socal sensng data. In prevous work, we developed a maxmum lkelhood estmator of relablty of both partcpants and facts concluded from the data. Ths paper presents an analytcally-founded bound that quantfes the accuracy of such maxmum lkelhood estmaton n socal sensng. A confdence nterval s derved by leveragng the asymptotc normalty of maxmum lkelhood estmaton and computng the approxmaton of Cramer-Rao bound (CRB) for the estmaton parameters. The proposed quantfcaton approach s emprcally valdated and shown to accurately bound the actual estmaton error gven suffcent number of partcpants under dfferent sensng topologes. 1. INTRODUCTION Socal sensng, where ndvduals act as sensors, s a key emergng category of sensng applcatons. Yet, quantfyng the relablty of data collected from human sources remans one of the man challenges n utlzng socal sensng n msson-crtcal systems. Ths relablty problem has long been known n mltary scenaros and s becomng ncreasngly mportant n commercal and cvl settngs as well. The man research queston s one of quantfyng a level of con- Permsson to make dgtal or hard copes of all or part of ths work for personal or classroom use s granted wthout fee provded that copes are not made or dstrbuted for proft or commercal advantage and that copes bear ths notce and the full ctaton on the frst page. To copy otherwse, to republsh, to post on servers or to redstrbute to lsts, requres pror specfc permsson and/or a fee. Ths artcle was presented at: 8th Internatonal Workshop on Data Management for Sensor Networks (DMSN 2011) Copyrght fdence n nformaton reported by a group of human observers. Ths paper quantfes confdence n socal sensng observatons by computng a confdence nterval of a maxmum lkelhood estmator of partcpant relablty from socal sensng data. The confdence nterval s computed based on the approxmaton of Cramer-Rao bound (CRB) [4, whch quantfes the varance of a mnmum-varance unbased estmator. Ths bound approxmates the CRB by leveragng observatons from partcpants and knowledge of truthfulness of facts estmated from an maxmum lkelhood estmator of socal sensng [14. We consder a sensng applcaton n whch data are collected from a large populaton, where the relablty (.e., the probablty of correctness) of ndvdual partcpants, and hence observatons made by them, s not known a pror. We am to derve the confdence nterval n the maxmum lkelhood estmaton of partcpant and observaton relablty gven no pror knowledge other than the nformaton descrbng who reported whch observatons. The maxmum lkelhood estmaton problem tself s not the topc of ths paper. An approach based on expectaton maxmzaton [5 s descrbed n pror work [14. Ths paper quantfes the confdence approxmately n the answer reported by such an estmator. We concern ourselves wth bnary measurements only; for example, reportng whether or not a gven person or object was seen at a gven locaton. Our dervaton leverages the asymptotc normalty of maxmum lkelhood estmaton and computes the approxmaton of Cramer-Rao bound (CRB) of the estmaton parameters used n an expectaton maxmzaton scheme. It s shown that the probablty of the estmated partcpant relablty fallng nto the derved confdence nterval s always greater than the gven confdence level, as long as enough partcpants report suffcent observatons, and as long as some partcpants make the same observaton. In other words, t s shown that our confdence wndow correctly bounds estmaton error. The rest of ths paper s organzed as follows: In Secton 2, we revew a maxmum lkelhood estmaton approach for socal sensng applcatons and formulate the problem of dervng the confdence nterval of partcpant relablty. The dervaton of the proposed approxmaton for CRB quantfcaton on maxmum lkelhood estmaton s dscussed n Secton 3. Evaluaton results are presented n Secton 4. We

2 revew related work n Secton 5. Fnally, we conclude the paper n Secton PROBLEM FORMULATION Frst, we revew the maxmum lkelhood estmaton approach for truth dscovery n socal sensng applcatons. We consder a socal sensng applcaton model where a group of M partcpants, S 1,..., S M, make ndvdual observatons about a set of N measured varables C 1,..., C N n ther envronment. Each measured varable denotes the exstence or lack thereof of certan phenomenon of applcaton s nterests. In ths effort, we consder only bnary varables and assume, wthout loss of generalty, that ther normal state s negatve. Hence, partcpants report only when a postve value s encountered. Each partcpant generally observes only a subset of all varables. Our goal s to determne the confdence nterval of the partcpant relablty maxmum lkelhood estmaton for a gven confdence level based only on the nformaton of whch observatons are reported by whch partcpant. Let S C j denote an observaton reported by partcpant S clamng that C j s true. Let P (Cj) t and P (C f j ) denote the probablty that the actual varable C j s ndeed true and false, respectvely. Dfferent partcpants may make dfferent numbers of observatons. Let the probablty that partcpant S makes an observaton be s. Further, let the probablty that partcpant S s rght be t and the probablty that t s wrong be 1 t. Note that, ths probablty depends on the partcpant s relablty, whch s not known a pror. Formally, t s defned as: t = P (C t j S C j) (1) Let us also defne a as the (unknown) probablty that partcpant S reports a varable to be true when t s ndeed true, and b as the (unknown) probablty that partcpant S reports a varable to be true when t s n realty false. Formally, a and b are defned as follows: a = P (S C j C t j) b = P (S C j C f j ) (2) From the defnton of t, a and b, we can determne ther relatonshp usng the Bayesan theorem: a = P (S C j Cj) t = P (SCj, Ct j) P (Cj t) = P (Ct j S C j)p (S C j) P (Cj t) b = P (S C j C f j ) = P (SCj, Cf j ) P (C f j ) = P (C f j SCj)P (SCj) P (C f j ) (3) The key nput to the maxmum lkelhood estmator algorthm s a matrx SC, where S C j = 1 when partcpant S reports that C j s true, and S C j = 0 otherwse. Let us call t the observaton matrx. For ntalzaton, we also defne the background bas d to be the overall pror probablty that a randomly chosen measured varable s true. Note that, ths value can be known from past statstcs. It does not ndcate, however, whether any partcular measured varable s true or not. To ntalze the algorthm, we set P (C t j) = d and set P (S C j) = s. Pluggng these, together wth t nto the defnton of a and b, we get the ntal values: t s a = d (1 t) s b = 1 d The best (n the sense of maxmum lkelhood) estmate ˆt of the relablty of each partcpant S can be obtaned by usng the Expectaton Maxmzaton (EM) algorthm [5. In the EM algorthm, a latent varable Z s ntroduced for each measured varable to ndcate whether t s true or not (.e., z j s 1 when the measured varable C j s true and 0 otherwse). The observaton matrx SC s treated as the observed data X, and θ = (a 1, a 2,...a M ; b 1, b 2,...b M ) s the parameter vector of the model we want to estmate. The EM algorthm teratvely performs an expectaton step (Estep) and a maxmzaton step (M-step) to compute the best estmate of the parameter θ that maxmzes the expected logarthm lkelhood functon. The lkelhood functon used by EM scheme s gven by: L(θ; X, Z) = p(x, Z θ) { N M = a S C j (1 a ) (1 S C j ) d z j + j=1 M =1 =1 b S C j (1 b ) (1 S C j ) (1 d) (1 z j) An output of the EM algorthm s the maxmum lkelhood estmaton of each partcpant s relablty, whch s most consstent wth the observaton matrx SC. However, an mportant problem that remans unanswered from the maxmum lkelhood estmaton of the EM scheme s: what s the confdence nterval of the resultng partcpant relablty estmaton? Only by answerng ths queston, can we completely characterze estmaton performance, and hence partcpant relablty n socal sensng applcatons. The goal of ths paper s to demonstrate, n an analytcally founded manner, how to compute the confdence nterval of each partcpant s relablty. Formally, ths s gven by: (ˆt MLE } (4) (5) c lower p, ˆt MLE + c upper p ) c% (6) where c% s the confdence level of the estmaton nterval, c lower p and c upper p represent the lower and upper bound on the estmaton devaton from the maxmum lkelhood estmaton ˆt MLE respetvely. We target to fnd c lower p and for a gven c% and an observaton Matrx SC. c upper p 3. RELIABILITY DERIVATION In ths secton, we derve a confdence nterval based on the approxmaton of Cramer-Rao Bound and the aforementoned formulaton of maxmum lkelhood estmaton of partcpant relablty. The log-lkelhood functon (or logprobablty densty functon) of the maxmum lkelhood estmaton we get from EM can be expressed as: l em(x; θ) = log p em(x; θ) { [ N M = z j (S C j log a + (1 S C j) log(1 a ) + log d) j=1 =1 + (1 z j) [ M } (S C j log b + (1 S C j) log(1 b ) + log(1 d)) =1 (7)

3 The lkelhood (or probablty densty functon) s: p em(x; θ) = exp(l em(x; θ)) (8) The goal here s to show that the confdence nterval of the estmated parameter θ can be asymptotcally characterzed by the approxmaton of Cramer-Rao bound (CRB) gven the observaton matrx as well as estmated truthfulness of each measured varable from EM scheme. In statstc mathematcs and nformaton theory, the Fsher nformaton s a way of measurng the amount of nformaton that an observable random varable X carres about an estmated parameter θ upon whch the probablty of X depends. The partal dervatve of the log-lkelhood functon l(x; θ) wth respect to θ s called the score. A score vector ψ(x; θ) for a k 1 estmaton vector θ = [θ 1, θ 2,..., θ k T s defned as: l(x; θ) l(x; θ) l(x; θ) ψ(x; θ) = [,,..., T (9) θ 1 θ 2 θ k The Fsher nformaton s defned as the second moment of the score vector: I(θ) = E X[ψ(X; θ)ψ(x; θ) T (10) where the expectaton s taken over all values for X wth respect to the probablty functon p(x; θ) for any gven value of θ. Hence, the Fsher nformaton for the above estmaton vector θ takes the form of an k k matrx, the Fsher Informaton Matrx, wth the representatve element: l(x; θ) l(x; θ) (I(θ)),j = E X[( )( ) (11) θ θ j If l(x; θ) s twce dfferentable wth respect to θ (whch happens to be the case for EM model), under certan regularty condtons, the Fsher Informaton Matrx may also be wrtten as [10: (I(θ)),j = E X[ 2 l(x; θ) θ θ j (12) In estmaton theory and statstcs, the Cramer-Rao bound (CRB) expresses a lower bound on the varance of estmators of a determnstc parameter. In ts smplest form, the bound states the varance of any unbased estmator s at least as hgh as the nverse of the Fsher nformaton [10. The estmator that reaches ths lower bound s sad to be effcent. The maxmum lkelhood estmator posses a number of attractve asymptotc propertes. One of them s called asymptotc normalty, whch bascally states the MLE estmator s asymptotcally dstrbuted wth Gaussan behavor as the data sample sze goes up, n partcular[3: n(ˆθmle θ 0) d N(0, I 1 (ˆθ MLE)) (13) where n s the sample sze, θ 0 and ˆθ MLE are the true value and the maxmum lkelhood estmaton of the parameter θ respectvely. The Fsher nformaton at the MLE s used to to estmate ts true value [10. Hence, the asymptotc normalty property means that n a regular case of estmaton and n the dstrbuton lmtng sense, the maxmum lkelhood estmator ˆθ MLE s unbased and ts covarance reaches the Cramer-Rao bound (.e., an effcent estmator). Snce the estmator we obtan from the EM algorthm s a maxmum lkelhood estmator of the parameter θ, we now show how to leverage the asymptotc normalty and the approxmaton of Cramer-Rao bound to derve a confdence nterval that quantfes the estmaton accuracy of the estmated parameter θ for the model of the EM scheme. We frst compute the approxmaton of Fsher Informaton Matrx from the log-lkelhood functon gven by Equaton (7). Note that ths computaton utlzes the estmated truthfulness of each measured varable from EM scheme, hence offers approxmated results. Accordng to pror work [14, the maxmum lkelhood estmator ˆθ MLE s gven by: â MLE = ˆbMLE j SJ Z c j N j=1 Zc j = K j SJ Z c j N N j=1 Zc j (14) where SJ s the set of measured varables reported by partcpant S and Zj c s the converged value of Z(t, j) (.e., p(z j = 1 X j, θ (t) )) from EM algorthm. Observe that each â MLE ˆb MLE or s computed from N ndependent samples (.e., measured varables). The Fsher nformaton n a random sample of sze n s n tmes the Fsher nformaton n one observaton [10, and hence I n(θ) = ni(θ) (15) Pluggng l em(x; θ) gven by Equaton (7) nto the Fsher Informaton Matrx defned n Equaton (12), we have: (I(ˆθ MLE)),j (16) 0 j [ = E 1 2 l em(x;a ) X N a =â MLE = j [1, M [ E 1 X N a 2 2 l em(x;b ) b 2 b =ˆb MLE = j (M, 2M Observe that the Fsher Informaton Matrx of the maxmumlkelhood estmator from the EM scheme s a dagonal matrx, hence the nverse of ths matrx s: (I 1 (ˆθ MLE)),j (17) 0 j [ E X = [ E X N 2 lem(x;a ) a 2 N 2 lem(x;b ) b 2 a =â MLE b =ˆb MLE = j [1, M = j (M, 2M From the asymptotc normalty of the maxmum lkelhood estmator specfed by Equaton (13), we know that (ˆθ MLE θ 0) d N(0, 1 N I 1 (ˆθ MLE)). Therefore, substtutng (I 1 (ˆθ MLE)) by Equaton (17) nto Equaton (13), we obtan the covarance matrx Cov(ˆθ MLE) of the asymptotc normal dstrbuton for the maxmum lkelhood estmaton of EM scheme, whch s gven by: (Cov(ˆθ MLE)),j (18) 0 j [ 1 E X = 2 lem(x;a ) a =â MLE = j [1, M a 2 [ 1 E X = j (M, 2M 2 lem(x;b ) b 2 b =ˆb MLE Usng the converged log-lkelhood functon of Equaton (7) and substtutng Equaton (14) nto Equaton (18), the above covarance matrx can be further wrtten as:

4 (Cov(ˆθ MLE)),j (19) 0 j â = MLE (1 â MLE ) = j [1, M N d ˆbMLE (1 ˆb MLE ) = j (M, 2M N (1 d) Note that, the actual CRB bound s a functon of both M and N. However, the approxmaton CRB bound derved s ndependent of M. Let us denote the varance of estmaton error on parameter a as V ar(â MLE ). Recall the relaton between partcpant relablty and estmaton parameter a s a = t s. For a gven topology, s d and d are known constants, (ˆt MLE t 0 ) also follows a norm dstrbuton wth 0 mean and varance gven by: V ar(ˆt MLE ) = ( d s ) 2 V ar(â MLE ) (20) Hence, we are able to derve the confdence nterval that can be used to quantfy the estmaton accuracy of the maxmum lkelhood estmaton from the EM scheme. The confdence nterval of the relablty estmaton of partcpant S (.e., ˆt MLE (ˆt MLE ) at confdence level p s gven by: c p V ar(ˆt MLE ), ˆt MLE + c p V ar(ˆt MLE )) (21) where c p s the standard score (z-score) of the confdence level p. For example, for the 95% confdence level, c p = Note that the derved confdence nterval of the partcpant relablty maxmum lkelhood estmator can be computed by smply usng the converged maxmum leklhood estmaton of the EM scheme. Ths completes the dervaton. 4. EVALUATION In ths secton, we carry out smulaton experments to evaluate the performance of the computed confdence nterval of partcpant relablty n socal sensng. We bult a smulator n Matlab that generates a random number of partcpants and measured varables. A random probablty P s assgned to each partcpant S representng hs/her relablty (.e., the ground truth probablty that they report correct observatons). For each partcpant S, L observatons are generated. Each observaton has a probablty P of beng true (.e., reportng a varable as true correctly) and a probablty 1 P of beng false (reportng a varable as true when t s not). One can thnk of these varables as observed problems. Partcpants do not report lack of problems. Hence, they never report a varable to be false. We let P be unformly dstrbuted between 0.5 and 1 n our experments 1. We evaluate the derved confdence nterval on partcpant relablty over three dfferent observaton matrx scales: small, medum and large. The smulaton parameters of the three observaton matrx scales are lsted n Table 1. The average observatons reported by each partcpant s set to 100. For each observaton matrx scale, we run the EM algorthm and compute the confdence nterval on partcpant relablty based on Equaton (21). We repeat the experments In prncple, there s no ncentve for a partcpant to le more than 50% of the tme, snce negatng ther statements would then gve a more accurate truth tmes for each observaton matrx scale and call the experments wth the actual estmaton error fallng outsde the confdence nterval outlers. We choose three representatve confdence levels (.e., 68%, 90%, 95% 2 ), respectvely. For a gven confdence level, we further defne the partcpant who has the largest number of outlers over all experments as the worst-case partcpant. Hence, we record the number of ourlters of every worst-case partcpant for a gven confdence level and compare t wth the theoretcal maxmum number of outlers. Observaton Number Number of Number of Matrx of Partcpantsured True Mea- False Measured Scale Var- ables Var- ables Small Medum Large Table 1: Parameters of Three Typcal Observaton Matrx Scale Fgure 1 shows the confdence nterval bounds on the partcpant relablty estmaton error wth three dfferent confdence levels for the small observaton matrx. Note that the CRB s smply a functon of the ground truth parameter values. However, t s reasonable to substtute the true (but unknown) parameter values wth ther ML estmates [10. Ths s the reason that bounds n the fgure appear to fluctuate rather than beng flat. Observe that the actual estmaton error on partcpant relablty s well bounded by ts correspondng confdence nterval. Specally, the numbers of outlers for the worst-case partcpant at confdence levels 68%, 90% and 95% are 24, 8 and 5 out of 100 experments. They are less than the theorectal maxmum number of outlers for the three confdence levels (.e., 32, 10 and 5 out of 100 experments, predcted usng our derved bound). Smlar results are observed for the medum and large observaton matrces as well, whch are shown n Fgure 2 and Fgure 3. A summary of the comparson between confdence nterval bounds n estmatng partcpant relablty and the theoretcal results s shown n Table 2. We observe that the probablty of the estmated partcpant relablty fallng nto the derved confdence nterval s always greater than the correspondng confdence level. Snce the derved bound s an approxmaton of true CRB (.e., t depends on the correct estmaton of truthfulness of measured varables from EM scheme), we study the condtons when such approxmated bound fals to bound the actual error of the estmaton parameters. We fx the true and false measured varables to be 1000 respectvely, the average observatons per partcpant s set to 100. We vary the number of partcpants from very small (.e., 5) to large (.e., 205). Reported results are averaged over 100 experments. Fgure 4 shows the square root of the average MSE (mean squared error) of 3 confdence nterval bounds on the estmated parameter a and b when the number of partcpants vares. Observe that the hgh confdence bounds (.e., 95% or 90% ) fal to bound the root of MSE on a or b only when the number of partcpants (M) s very small. Ths s due to 2 They correspond to one, two and three tmes standard devaton confdence ntervals of normal dstrbuton

5 (a) 68% Confdence Bound (b) 90% Confdence Bound (c) 95% Confdence Bound Fgure 1: CRB Confdence Bounds on Partcpant Relablty for Small Observaton Matrx (a) 68% Confdence Bound (b) 90% Confdence Bound (c) 95% Confdence Bound Fgure 2: CRB Confdence Bounds on Partcpant Relablty for Medum Observaton Matrx (a) 68% Confdence Bound (b) 90% Confdence Bound (c) 95% Confdence Bound Fgure 3: CRB Confdence Bounds on Partcpant Relablty for Large Observaton Matrx (a) Confdence Bound on a (b) Confdence Bound on b Fgure 4: CRB Confdence Bound on a and b versus Vayng M the poor estmaton results of measured varbles when too few partcpants report ther observatons. However, when M s reasonablly suffcent (e.g., 25 n the experment), hgh confdence bounds always bound the square root of the average MSE on the estmated parameters correctly. 5. RELATED WORK Socal sensng has receved sgnfcant attenton due to the great ncrease n the number of moble sensors owned by ndvduals and the prolferaton of Internet connectvty. To assess the credblty of partcpants and facts reported n partcpatory sensng and other socal sensng applcatons, a relevant body of work, called fact-fnders, n the machne learnng and data mnng communtes performs trust analyss. The basc fact-fnders nclude Hubs and Authortes [11,

6 Observaton Matrx Scale Small Medum Large Confdence Level of Estmaton Theoretcal Maxmum Outlers/Total Experments 68% 32/100 24/100 90% 10/100 8/100 95% 5/100 5/100 68% 32/100 25/100 90% 10/100 6/100 95% 5/100 4/100 68% 32/100 25/100 90% 10/100 9/100 95% 5/100 2/100 CRB Bound Worst Case Outlers/Total Experments Table 2: CRB Bound on Partcpant Probablty versus Theoretcal Results Average.Log [12, and TruthFnder [15. Other fact-fnders enhance the basc framework by ncorporatng analyss on propertes or dependences wthn assertons or sources [9, 2, 8, 7, 6. Fact-fndng n the case of socal sensng s more challengng due to the hghly dynamc nature of socal sensng applcatons [1. Moreover, the outputs of fact-fnders are generally rankngs of credblty values of partcpants and facts. Such rankngs cannot be used to drectly quantfy the partcpant relablty or fact correctness. Recent work presented a Bayesan Interpretaton scheme [13 representng an ntal effort to provde a probablty nterpretaton of rankng outputs from fact-fnders. However, t remans an approxmaton approach n whch the accuracy of truth estmaton s very senstve to ntal condtons of teratons. Due to ths lmtaton, a maxmum lkelhood estmaton approach usng EM algorthm s proposed to provde the frst optmal soluton to the truth dscovery problem n socal sensng [14. The EM scheme was shown to outperform Bayesan nterpretaton and other state-of-art factfnders. However, only average estmaton accuraces were reported n both of above schemes. The confdence nterval of the estmaton accuracy has not been found. In contrast, ths paper derves, for the frst tme, the confdence nterval of partcpant relablty based on the approxmaton of CRB, hence completes the quantfcaton of partcpant relablty estmaton n socal sensng. In estmaton theory and statstcs, the Cramer-Rao bound refers to a lower bound on the varance of estmators of a determnstc parameter [4. The bound states the varance of any unbased estmator s lower-bounded by the nverse of Fsher nformaton [10. One of the key propertes of maxmum lkelhood estmaton s asymptotc normalty. An EM scheme provdes maxmum lkelhood estmaton of partcpant relablty for socal sensng applcatons. Ths paper provdes the frst quantfcaton approach to compute the confdence nterval for partcpant relablty maxmumlkelhood estmaton based on the approxmaton of CRB by leveragng results from estmaton and nformaton theory. 6. CONCLUSION Ths paper descrbed a quantfcaton approach to compute the confdence nterval of the maxmum lkelhood estmaton on partcpant relablty based on the approxmaton of CRB n socal sensng applcatons. Ths quantfcaton approach completely characterzes the estmaton performance of partcpant relablty wthout knowng the trustworthness of partcpants a pror. The derved confdence nterval s obtaned by leveragng the asymptotc normalty of maxmum lkelhood estmaton and can be easly computed from the approxmated Fsher nformaton contaned n partcpants relablty estmatons. Evaluaton results show that the error n the estmated partcpant relablty s well bounded by the computed bound. In future work, a tghter CRB bound can probably be derved wthout knowng the truthfulness of each measured varable. Ths actual CRB bound s expected to better track the actual MSE of the estmated parameters. Acknowledgements Research reported n ths paper was sponsored by the Army Research Laboratory and was accomplshed under Cooperatve Agreement Number W911NF The vews and conclusons contaned n ths document are those of the authors and should not be nterpreted as representng the offcal polces, ether expressed or mpled, of the Army Research Laboratory or the U.S. Government. The U.S. Government s authorzed to reproduce and dstrbute reprnts for Government purposes notwthstandng any copyrght notaton here on. 7. REFERENCES [1 C. Aggarwal and T. Abdelzaher. Integratng sensors and socal networks. Socal Network Data Analytcs, Sprnger, expected n [2 L. Bert-Equlle, A. D. Sarma, X. Dong, A. Maran, and D. Srvastava. Salng the nformaton ocean wth awareness of currents: Dscovery and applcaton of source dependence. In CIDR 09, [3 G. Casella and R. Berger. Statstcal Inference. Duxbury Press, [4 H. Cramer. Mathematcal Methods of Statstcs. Prnceton Unv. Press., [5 A. P. Dempster, N. M. Lard, and D. B. Rubn. Maxmum lkelhood from ncomplete data va the em algorthm. JOURNAL OF THE ROYAL STATISTICAL SOCIETY, SERIES B, 39(1):1 38, [6 X. Dong, L. Bert-Equlle, Y. Hu, and D. Srvastava. Global detecton of complex copyng relatonshps between sources. PVLDB, 3(1): , [7 X. Dong, L. Bert-Equlle, and D. Srvastava. Truth dscovery and copyng detecton n a dynamc world. VLDB, 2(1): , [8 X. L. Dong, L. Bert-Equlle, and D. Srvastava. Integratng conflctng data: the role of source dependence. Proc. VLDB Endow., 2: , August [9 A. Galland, S. Abteboul, A. Maran, and P. Senellart. Corroboratng nformaton from dsagreeng vews. In WSDM, pages , [10 R. V. Hogg and A. T. Crag. Introducton to mathematcal statstcs. Prentce Hall, [11 J. M. Klenberg. Authortatve sources n a hyperlnked envronment. Journal of the ACM, 46(5): , [12 J. Pasternack and D. Roth. Knowng what to beleve (when you already know somethng). In Internatonal Conference on Computatonal Lngustcs (COLING), [13 D. Wang, T. Abdelzaher, H. Ahmad, J. Pasternack, D. Roth, M. Gupta, J. Han, O. Fatemeh, and H. Le. On bayesan nterpretaton of fact-fndng n nformaton networks. In 14th Internatonal Conference on Informaton Fuson (Fuson 2011), [14 D. Wang, T. Abdelzaher, and L. Kaplan. On truth dscovery n socal sensng: A maxmum lkelhood estmaton approach. UIUC Techncal Report, [15 X. Yn, J. Han, and P. S. Yu. Truth dscovery wth multple conflctng nformaton provders on the web. IEEE Trans. on Knowl. and Data Eng., 20: , June 2008.