arxiv: v1 [cs.lg] 8 Jan 2019

Transcription

1 Data Masking with Privacy Guarantees Anh T. Pha Oregon State University Shalini Ghosh Sasung Research Vinod Yegneswaran SRI international arxiv:90.085v [cs.lg] 8 Jan 09 Abstract We study the proble of data release with privacy, where data is ade available with privacy guarantees while keeping the usability of the data as high as possible this is iportant in health-care and other doains with sensitive data. In particular, we propose a ethod of asking the private data with privacy guarantee while ensuring that a classifier trained on the asked data is siilar to the classifier trained on the original data, to aintain usability. We analyze the theoretical risks of the proposed ethod and the traditional input perturbation ethod. Results show that the proposed ethod achieves lower risk copared to the input perturbation, especially when the nuber of training saples gets large. We illustrate the effectiveness of the proposed ethod of data asking for privacy-sensitive learning on benchark datasets. Introduction In doains like healthcare or finance, data can be sensitive and private. There are several scenarios where a dataset needs to be shared while protecting sensitive parts of the data. For exaple, consider a edical study where a group of patients with a particular edical condition are being studied. The identifying data of soe patients (e.g., those with a rare disease) ay need to be asked while sharing their records with a wider group of edical researchers. However, when the patient records are processed by clinical decision support tools, we want the achine learning (ML) odels in the tools to have siilar perforance on the asked data as they would on the original data. Several approaches have been proposed to preserve privacy of data, e.g., by anonyization (Saarati and Sweeney 998), by generalization (Mohaed et al. 0). Methods for differential-privacy include adding Laplace-noise (Sarwate and Chaudhuri 03), odifying the objective (Chaudhuri and Monteleoni 009), and posterior sapling (Diitrakakis et al. 04; Wang, Fienberg, and Sola 05). Privacy-preserving data publishing transfors sensitive data to protect it against privacy attacks while supporting effective data ining tasks (Fung et al. 00). Differentially private data release (Mohaed et al. 0) presents an anonyization algorith that satisfies the ɛ differential privacy odel, while other ethods of data release (Chen et al. 0; Xiao, Xiong, and Yuan 00) group the data and add noise to the partition counts. However, these techniques don t explicitly try to aintain the accuracy of a odel. Our approach asks training saples with less sensitive ones with privacy guarantee, while ensuring that the classifier trained on the asked data reaches accuracy siilar to the classifier trained on the original data. Moreover, copared to publishing asked classifier, publishing asked data enables other types of classifiers to be trained by the user. There are also query-based data asking ethods for a classifier, which are sparse vector techniques for generating asked data using a query that the gradient of the asked data is zero (Dwork, Roth, and others 04; Lyu, Su, and Li 07; Lee and Clifton 04; Blu, Ligett, and Roth 008). However, when the gradient coputation is coplicated, designing a ethod to achieve a zero gradient can be tricky. We have three ain contributions in this paper. First, we propose a novel algorith of data asking for privacysensitive learning. Second, we provide a theoretical guarantee explaining why the proposed ethod is ore suitable for a large nuber of training saples than a traditional input perturbation ethod. Finally, we illustrate the efficacy of our ethod considering logistic regression as an exaple classifier, on both synthetic and benchark datasets. Proble setting Goal: Assue we train a odel paraeterized by w R d on a dataset D train = {x i, y i } N i=, where x i X = R d, y i Y = {, }, and d is the nuber of features. The goal of our data publishing algorith A is generating a asked training dataset D asked = {x i, y i} N i=, where x i X, such that: (a) D asked is as different as possible fro D train, but (b) the odel trained on D asked gives us paraeters w that are close to the original paraeters w of the odel trained on D train. This paper outlines an approach for achieving this goal. Before that, we review several concepts of data publishing with privacy and the core forulation of logistic regression. Data publishing with differential privacy (DPDP) We first begin with the concept of data publishing with differential privacy (DPDP). We consider two datasets of N training saples, D train = {x i, y i } N i= and D train =

2 {x i, y i } N i=, which are different at only one saple: without loss of generality, assue x i = x i and y i = y i for i = {,,..., N }, and x N x N and (or) y N y N. A data publishing algorith A is said to be ɛ-private (Dwork 008) if p ( A(D train ) = O ) p(a ( D train ) = O ) < eɛ, where O = {x i, y i} N i= is a particular output of the data publishing algorith A. Intuitively, differential privacy guarantees that for sall ɛ, the output of A is not sensitive to the existence of a single saple in the dataset. In this setting, the attacker has less chance to infer details about a particular training saple in the data. In this work we focus on differential privacy for asked data generation where the achine learning algorith we consider is logistic regression (Walker and Duncan 967). Core forulation of logistic regression Logistic Regression: We are given a training dataset D train. The goal for training a logistic regression classifier is finding a apping function between a saple in R d and a label in {, }. Specifically, we odel the relation aong a saple x i and its label y i as p(y i x i, w) = eyiwt x i + e yiwt x i. Assuing saples are i.i.d., the log-likelihood for the training saples is L λ (w) = N [ (yi w T ) x i log(+e y iw T x i )]+λ w, N i= () with λ is the regularization paraeter and denotes the -nor. Training logistic regression: In logistic regression, training is done by finding the paraeter w that axiizes the loglikelihood in (), i.e., the gradient of L λ at w is 0, as follows: L λ (w) w [ N [ ] y i p(y i = x i, w) ]x i + λw = 0. = N i= () For various logistic regression optiization techniques to ake the above gradient 0, please refer to (Minka 003). DPDP by Masked Data Generation In this section, we describe how to generate asked saples for logistic regression. Adding Laplace noise to the classifier Unlike previous approaches of adding noise to the data then publishing noisy data, we consider a novel approach: we first In this work we consider logistic regression as the classifier. The work flow of data publishing for other classifiers, e.g., SVM, is siilar to that of the proposed ethod. {x, y } {x, y } {x N, y N } {x N, y N } Logistic Regression Laplace noise {x, y } {x, y } {x, y } {x N, y N } {x N, y N } w Laplace noise w (a) The proposed asked data generation {x, y } {x N, y N } {x N, y N } (b) The input perturbation ethod Masked Data Generation {x, y } {x, y } {x N, y N } {x N, y N } Figure : The work flow of the proposed ethod and traditional data publishing ethods. train a classifier on the original data, and then add Laplace noise to the classifier. The otivation for adding noise is that in differential privacy, the goal is to ake siilar output for any two neighbor datasets D and D so that attacker cannot infer about the existence of any single training saple. Since the classifiers trained on two datasets D and D are not equal, adding Laplace noise to the paraeters of those classifiers would account for that difference, and with soe probability those classifiers after adding noise would be equal. Subsequently, we generate and publish a asked dataset such that the gradient of the log-likelihood for the noisy classifier is 0. The work flow of the proposed fraework is illustrated in Fig. (a). In coparison, the work flow of traditional data publishing ethods by perturbation is shown in Fig. (b). Generating asked data We generate asked data O = {x i }N i= such that the gradient of the log-likelihood of O for the aforeentioned noisy classifier w is 0. The optial condition for asked data is the following: N [ N i= [ ] ] y i p(y i = x i, w ) x i + λw = 0, (3) where the asked saples {x i, y i }(s) are unknown. To evaluate the optiality of the set S of asked saples w.r.t. w, we use the -nor of the gradient: N ( ) N (S) = y i p(y i = x i, w )x i + Nλw i= We start with an initial set of training saples S then iteratively add new saple to S. The criteria to evaluate the new saple is the -nor of the gradient of S after including the new saple. Algorith outlines our proposed Masked Data Generation algorith. The algorith terinates when the nuber of saples in S reaches N.

3 Algorith Masked Data Generation Input: N training saples D train = {x i } N i=, ɛ Output: N asked training saples O = {x i }N i=. Step : Train Logistic regression classifier w, as in (). Step : Add Laplace noise to the classifier w = w + η, where η c e λnɛ η, where c is a noralized constant. Step 3: S = { }. Increentally generate asked saples. while cardinality(s) N do Find an outliers {x } reducing the -nor of the gradient of S the ost, using Gradient Descent (5) Add the new saple S = S {x } end while Return O = S; Iteratively generating asked saples In this section, we present the gradient descent ethod to iteratively generate asked saples. In particular, given the current set of asked saples S, we need to find the next asked saple {x } such that the -nor of the gradient of the set S {x } is close to 0 as possible. For siplicity of notation, denote x (y i S i p(y i = ) x i, w)x i + Nλw = g as the current gradient of the current asked saples. Consequently, we need to find the next asked saple {x } iniizing the following objective N (x ) = ( x y eywt ) + e wt x x + g. (4) To iniize (4), we use backtracking gradient descent. The gradient is coputed as N (x ( ) (y x x = eywt ) + e wt x x + g) x ((y ewt )I w T x e wt x ) + e wt x, (5) ( + e wt x ) where I is the identity atrix in R d d. Note that, we can generalize our algorith to C classes, with C >, as follows N x = C c= ) ([I(y = c) p(y = c x, w)]x + g c ( [I(y = c) p(y = c x, w)] C ) [p(y = c x, w)p(y = l x, w)(w c w l ) T x )]. l= Coputational coplexity: The coputational coplexity of the proposed algorith is linear in ter of nuber of added saples. Intuition: Most differential privacy algoriths for data publishing odify the data by adding unifor noise, e.g., as in Fig. (b), which ay change the original data anifold closer to a unifor anifold and ay not be optiized for any particular achine learning odel. Coparison to classifier publishing: The proposed approach has an advantage over other traditional approaches. In particular, assuing a non-epty initialized set of training saples S in Step 3 of Algorith, the proposed ethod adds fake saples with copletely different anifold to the dataset. For exaple, assue we want to preserve the privacy of a dataset consisting of non-diabetes patients and sensitive type- diabetes patients. We can initially add nonsensitive type- diabetes data saples, thereby preserving the privacy of the type- diabetes patients. Moreover, by iteratively adding asked saples, a classifier that is trained on the original data will be quite close to the classifier trained on the new asked data. Copared to publishing the noisy classifier as in (Chaudhuri and Monteleoni 009), the proposed data asking ethod allows users to benefit fro real data, i.e., in this case non-diabetes and type- diabetes data, and train other types of classifiers on the. Privacy guarantee of Masked Data Generation There are two aspects of a data publishing algorith. First, we need to guarantee that the algorith is ɛ-private. In particular, is the algorith sensitive to the existence of a single saple in two datasets that are different only at that saple? Second, we would like to assess how the utility of the published dataset changes with changing ɛ. The following Proposition answers the first question. Proposition If x i, i, then Algorith is ɛ- private. Utility of Masked Data Generation with changing ɛ We next consider the utility aspect of the asked dataset O with different values of ɛ. We consider the utility of the published data to be how well the classifier trained on the published data is close to the classifier trained on the original data. Let us suppose that training logistic regression on the original dataset D train and the asked dataset O gives us paraeters w and w, respectively. We are interested in coparing the 0/ risk (Vapnik and Vapnik 998) of the classifier trained on asked data (w ), to the 0/ risk of the classifier trained on original data (w). Note that logistic regression is classification calibrated (Bartlett, Jordan, and McAuliffe 006), which eans that iniizing the negative log-likelihood leads to iniizing the 0/ risk. Thus, it is sufficient to copare the log-likelihood L λ of w copared to that of w. Proposition With probability δ, L λ (w ) L λ (w) ( d log( d δ ) λnɛ ) (λ + ). Fro Lea, the classifier trained on asked data iproves when N is larger.

4 DPDP by Input Perturbation In this section, we consider a classical and natural algorith to publish data (Sarwate and Chaudhuri 03; Mivule 0). The algorith is quite siple: it directly adds noise η e ɛ η to each input saple. The detailed algorith is shown in Algorith. Siilar to Algorith, in the rest of this section we consider the privacy and the utility of the input perturbation algorith when ɛ changes. Privacy guarantee of Input Perturbation We first show that Algorith is ɛ-private. Proposition 3 If x i, i, then algorith is ɛ-private. Algorith Input Perturbation Input: N training saples D train = {x k } N k=, ɛ Output: N asked training saples O = {x k }N k= while k < N do η e ɛ η x k = x k + η k = k + end while Return O = {x k }N k= Utility of Input Perturbation with changing ɛ Siilar to Section, we consider the log-likelihood of the classifier w trained on perturbed data. We are going to bound the log-likelihood w.r.t. the original data L λ (w ) L λ (w). We begin with the following Proposition. Lea 4 (Chaudhuri and Monteleoni 009). Let G(w) be a convex function and g(w) be a function with g(w) g and in v in w v T (G + g)(w)v G. Let w = arg in G(w) and w = arg in G(w) + g(w). Then w w g G. Proposition 5 With probability δ, L λ (w ) L λ (w) ( d log d δ λɛ ) (λ + ). Fro Proposition 5, the classifier trained on perturbed data does not iprove when N is larger, as we see in Proposition. Experients We copare the perforance of our Masked Data Generation ethod in Algorith to the Input Perturbation ethod in Algorith, on both synthetic and real datasets. Results on toy data Datasets: In this section, the effectiveness of the proposed ethod is illustrated on a D toy dataset. We saple 00 training saples fro three noral distributions. The st class coes fro N ([0;.5] T, 0.5I), the nd class coes fro N ([; ] T, 0.5I), and the 3rd class coes fro N ([; ] T, 0.5I), as shown in Fig. (a). Assue that saples fro the 3rd class is sensitive. Setting: We initialize the saples in the asked dataset fro a class with a different anifold for the 3rd class. In particular, we first add to the published dataset a fake class 3 with a totally different distribution anifold fro the original class 3, e.g., N ([; ] T, 0.5I) instead of N ([; ] T, 0.5I), as shown in Fig. (b). We then run the asked data generation ethod with non-epty training saples set S as in Algorith. Results: The saples generated fro the proposed ethod are shown in Fig. (c). Fro Fig. (c), to accoodate for the shift in distribution anifold of class 3 fro [; ] to [; ], any other fake saples of class 3 are added in the botto of Fig. (c).fro Fig. (c), we observe the usefulness of regularization, since less asked saples are on the boundary. Fro Fig. (c) and Fig. (a), the generated saples fro class 3 is significantly different fro the original true saples fro class 3, which iplies that the data is private. However, the resulting classifier or the boundary learned fro the three classes are alost siilar for original data and published data. As a result, users are still able to access original real data fro classes and, and at the sae tie achieve the classifier for class 3 which is private now. Results on MNIST digits data In this section, we consider the effectiveness of the proposed algorith on the MNIST handwritten digit dataset. Datasets: We use PCA to reduce the diensionality of the data to 5. Siilar to the toy exaple, we select saples fro three digits, e.g., three digits {0,, 3} as in Fig. 3(a), and three digits {0,, 4} as in Fig. 4(a). The corresponding classifier learned fro three digits {0,, 3} is shown in Fig. 3(d), and fro three digits {0,, 4} is shown in Fig. 4(d). Fro those figures, e.g., in Fig. 3(d), the visualized classifier represents the three corresponding digits {0,, 3}. Setting: We first explain how to generate a non-epty initially asked training saples S in Algorith. In particular, the first two digits fro the initially asked training saples are the sae as the two digits of the original training saples. For exaple, we still uses saples fro digits 0 and for initially asked training saples as in Fig. 3(b). However, for the last digit of the initially asked training saples, we use a totally different digit fro the last digit of the true training saples. For exaple, we use digit 6 instead of digit 3 as the last digit as in Fig. 3(b). The corresponding classifier learned fro the initially asked training saples S is visualized in Fig. 4(e). For visualization of a classifier, e.g., in Fig. 3(d-f), we project the classifier of each class back to the two diensional space.

5 (a) True training saples -4-4 (b) Initially asked saples set S -4-4 (c) Final asked saples Figure : (a) Original training saples, (b) Initially asked training saples, (c) Final asked saples using Algorith (a) True training saples 3 3 (b) Initially asked saples set S (c) Final asked saples using Algorith (d) True training w (e) Initially asked w set S (f) Masked w Figure 3: (a) True training saples, (b) Initially asked saples in S, (c) Final asked saples using Algorith, and their corresponding w s visualization (d-f) on MNIST datasets. We have digits fro 0,, 3 and we would like to replace 3 with 6 using soe fake saples (a) True training saples 3 3 (b) Initially asked saples set S (c) Final asked saples using Algorith (d) True training w (e) Initially asked w (f) Masked w Figure 4: (a) True training saples, (b) Initially asked saples in S, and (c) Final asked saples using Algorith, and their corresponding w s visualization (d-f) on MNIST datasets. We have digits fro 0,, 4 and we would like to replace 4 with 8 using soe fake saples.

6 Result: We then iteratively add asked training saples into S using the asked data generation ethod in Algorith. The asked saples generated by Algorith into S are shown in Fig. 3(c). Note that several saples aong the reove the effect of digit 6, e.g., the 4th saple fro the left in the first row of Fig. 3(c). On another hand, several aong the add the effect of digit 3 back to the classifier, e.g., the iage at the botto right of Fig. 3(c). Moreover, because of the adding asked saples, the classifier learned fro the asked training saples is siilar to the original classifier learned fro the original training saples. For exaple, the classifier in Fig. 3(f) is siilar to the classifier in Fig. 3(d). A siilar visualization exaple is shown in Fig. 4, where the original training saples are digit {0,, 4} as in Fig. 4(a), the initially asked training saples in S are digit {0,, 8} as in Fig. 4(b), and after generating asked saples, the classifier of the asked data as in Fig. 4(f) is siilar to the classifier of original data as in Fig. 4(d). Results on UCI datasets Datasets: We deonstrate the effectiveness of the proposed ethod on several UCI datasets in sensitive doains. Evaluation Measure: For all datasets, we uniforly select a validation set V = {x i } n val i= of saples fro two classes. We denote the ground truth labels for these saples as L true. Using w, the classifier trained on asked data, we predict the labels for the validation set, naely V asked. Then, we copute the accuracy of w as the fraction of cases where V asked atches L true. Setting: We consider the regularization paraeter λ =. Moreover, to evaluate the effectiveness of the proposed ethod and the input perturbation ethod when the nuber of training saples increases, we consider two cases: N = 00 and N = 00. We vary the value of ɛ in the set {0., 0.,,,,..., 0, 50}, e.g., log-scale. For each value of ɛ, we generate 50 training datasets, run the proposed asked data generation Algorith and the input perturbation Algorith on each dataset, then report the ean and standard deviation accuracy of both algoriths. We also evaluate the accuracy using the classifier after adding Laplace noise, i.e., after Step of Algorith, which is naed as output perturbation. Analysis: As shown in Fig. 5, first, as ɛ increases, the accuracy of both ethods increase. Additionally, for a particular value of ɛ, the proposed ethod works better than input perturbation algorith. Moreover, as N increases fro 00 to 00, the proposed ethod gets higher accuracy for the sae value of ɛ. In contrast, the accuracy of the input perturbation ethod does not change uch as N increases. Furtherore, note that the input perturbation ethod only updates the data independently fro the achine learning odel. In contrast, the data generated by the proposed ethod is directly tied to the odel, e.g., logistic regression with a particular value of λ, which ay lead to higher accuracy. Moreover, the perforance of the classifier trained on asked saples is coparable to those of the classifier trained on original training saples then adding Laplace noise, i.e., after Step of Algorith. The results indicate that the proposed asked data generation Algorith is able to create asked saples with corresponding classifier close to the perturbed classifier. Conclusions In this paper, we proposed a data asking technique for privacy-sensitive learning. The ain idea is to iteratively find asked data such that the gradient of the likelihood on the classifier with regarding to the asked data is zero. Our theoretical analysis showed that the proposed technique achieves higher utility copared to a traditional input perturbation technique. Experients on ultiple real-world datasets also deonstrated the effectiveness of the proposed ethod. Appendices Proof for Proposition. Assue there are two training datasets D = {x i } N i= and D = {x i } N i=, which are different at only one saple. Without the loss of generality, we assue x i = x i for i = {,,..., N }, and x N x N. Assue the outputs of Algorith is O = {x, x,..., x p(o D) N }. Consider the ratio p(o D ). We assue that in Step 3 we can find the output O such that the gradient of logistic regression objective w.r.t. w is exactly 0. For the classifier in Step, we consider w = a for the first dataset D and w = a for the second dataset D. Using the fact that the log-likelihood of logistic regression is convex, and a and a are both optial classifiers of the published data O, thus a = a = a. Then, the ratio p(o D) p(o D ) is coputed as: p(o D ) p(o D ) = p(o w = a)p(w = a D ) p(o w = a)p(w = a D ) = p(w = a D ) p(w = a D ). Assue w = b and w = b are the optial classifiers for D and D after Step. Therefore, because of Laplace noise in Step, b + η = b + η = a p(w =a D ) λnɛ = e ( η η ). Consequently, p(w =a D = p(η=η) ) p(η=η ) p(o D ) p(o D e λnɛ ( b b ) ) e λnɛ ( b b ). The sensitivity of logistic regression with N saples and regularization paraeter λ is atost λn (Chaudhuri and Monteleoni 009) p(o D) p(o D ) eɛ, which copletes the proof. Proof for Proposition. Since w is achieved fro w by adding Laplace noise, w w is bounded. So, L λ (w ) L λ (w) is bounded using Taylor series. The rest of the proof follows fro Lea in (Chaudhuri and Monteleoni 009). Proof for Proposition 3. Assue there are two training datasets D = {x i } N i= and D = {x i } N i=, which are different at only one saple, e.g., without the loss of generality, we assue x i = x i for i = {,,..., N }, and x N x N. Assue the outputs of of Algorith is

7 (a) Adult incoe (b) Geran credit 0. (c) Age of Abalone. (d) Wave for (e) AUS credit 0. (f) Breast cancer (g) Blood transfusion 5 (h) Heart disease 5 (k) Diabetics (i) Maographic (j) SpliceDNA 5 () Iage Figure 5: The accuracy and privacy (ɛ in log-scale) trade off for benchark datasets.

8 O = {x, x,..., x N, x N }. Consider the ratio p(o D ) p(o D ) = p(x x )... p(x N x N )p(x N x N) p(x x )... p(x N x N )p(x N x N) = e ɛ x N x N e ɛ x N x N eɛ x N x N e ɛ, where the last equation is fro the fact that x i, i. Thus, the input perturbation algorith is ɛ-private. Proof for Proposition 5. The proof is siilar to (Chaudhuri and Monteleoni 009). For the sake of copleteness, following Lea 4, define G(w) = H (w) + λ w and g = H (w) H (w), where H (w) = [ N (yi ) N i= w T x ] C i log( x l= ewt i ) and H (w) = [ N (yi ) N i= w T ] C x i log( x i l= ewt ). [ N ( Then, g = N i= (y i p(y i = ) x i, w) x i + N i= ( ] (y i p(y i = x i, w) )x i N N i= x i x i (Nd log d δ )/(Nɛ) = d log d δ /ɛ, where the last inequality coes fro the fact x i x i e ɛ η and x i, x i R d. Note that even though x i is upper bounded by i, x i is not upper bounded by since x i = x i + η where η e ɛ η. Hence, g can not be trivially upper bounded by. Moreover, v T (G + g)v is lower bounded by λ. Thus, w w d log d δ λɛ. By Taylor expansion, L λ (w ) = L λ (w) + L λ (w)(w w) + (w w) T L λ (w)(w w) L λ (w) + w w (λ + ). This copletes the proof. Acknowledgents This aterial is based upon work supported by the National Science Foundation under Grant CNS References [Bartlett, Jordan, and McAuliffe 006] Bartlett, P. L.; Jordan, M. I.; and McAuliffe, J. D Convexity, classification, and risk bounds. Journal of the Aerican Statistical Association 0(473): [Blu, Ligett, and Roth 008] Blu, A.; Ligett, K.; and Roth, A A learning theory approach to noninteractive database privacy. In Proceedings of the fortieth annual ACM syposiu on Theory of coputing, [Chaudhuri and Monteleoni 009] Chaudhuri, K., and Monteleoni, C Privacy-preserving logistic regression. In Advances in neural inforation processing systes, [Chen et al. 0] Chen, R.; Mohaed, N.; Fung, B. C.; Desai, B. C.; and Xiong, L. 0. Publishing set-valued data via differential privacy. In Proceedings of the International Conference on Very Large Data Bases, nuber, [Diitrakakis et al. 04] Diitrakakis, C.; Nelson, B.; Mitrokotsa, A.; and Rubinstein, B. 04. Robust and private bayesian inference. In Proceedings of the International Conference on Algorithic Learning Theory, [Dwork, Roth, and others 04] Dwork, C.; Roth, A.; et al. 04. The algorithic foundations of differential privacy. Foundations and Trends R in Theoretical Coputer Science 407. [Dwork 008] Dwork, C Differential privacy: A survey of results. In International Conference on Theory and Applications of Models of Coputation, 9. Springer. [Fung et al. 00] Fung, B.; Wang, K.; Chen, R.; and Yu, P. S. 00. Privacy-preserving data publishing: A survey of recent developents. ACM Coputing Surveys (CSUR) 4(4):4. [Lee and Clifton 04] Lee, J., and Clifton, C. W. 04. Topk frequent itesets via differentially private fp-trees. In Proceedings of the International Conference on Knowledge Discovery and Data Mining, [Lyu, Su, and Li 07] Lyu, M.; Su, D.; and Li, N. 07. Understanding the sparse vector technique for differential privacy. In Proceedings of the International Conference on Very Large Data Bases, [Minka 003] Minka, T. P A coparison of nuerical optiizers for logistic regression. [Mivule 0] Mivule, K. 0. Utilizing noise addition for data privacy, an overview. In Proceedings of the International Conference on Inforation and Knowledge Engineering (IKE),. [Mohaed et al. 0] Mohaed, N.; Chen, R.; Fung, B.; and Yu, P. S. 0. Differentially private data release for data ining. In Proceedings of the International Conference on Knowledge Discovery and Data Mining, [Saarati and Sweeney 998] Saarati, P., and Sweeney, L Generalizing data to provide anonyity when disclosing inforation. In PODS, 88. [Sarwate and Chaudhuri 03] Sarwate, A. D., and Chaudhuri, K. 03. Signal processing and achine learning with differential privacy: Algoriths and challenges for continuous data. IEEE signal processing agazine 30(5): [Vapnik and Vapnik 998] Vapnik, V. N., and Vapnik, V Statistical learning theory, volue. Wiley New York. [Walker and Duncan 967] Walker, S. H., and Duncan, D. B Estiation of the probability of an event as a function of several independent variables. Bioetrika 54: [Wang, Fienberg, and Sola 05] Wang, Y.-X.; Fienberg, S.; and Sola, A. 05. Privacy for free: Posterior sapling and stochastic gradient onte carlo. In Proceedings of the International Conference on Machine Learning, [Xiao, Xiong, and Yuan 00] Xiao, Y.; Xiong, L.; and Yuan, C. 00. Differentially private data release through ultidiensional partitioning. Secure Data Manageent 6358:50 68.