Knockoffs as Post-Selection Inference

Transcription

1 Knockoffs as Post-Selection Inference Lucas Janson Harvard University Department of Statistics blank line blank line WHOA-PSI, August 12, 2017

2 Controlled Variable Selection Conditional modeling setup: Y an outcome of interest (AKA response or dependent variable), X 1,..., X p a set of p potential explanatory variables (AKA covariates, features, or independent variables), Want to learn about the conditional distribution of Y given X 1,..., X p Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 1 / 15

3 Controlled Variable Selection Conditional modeling setup: Y an outcome of interest (AKA response or dependent variable), X 1,..., X p a set of p potential explanatory variables (AKA covariates, features, or independent variables), Want to learn about the conditional distribution of Y given X 1,..., X p How can we select important explanatory variables with few mistakes? Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 1 / 15

4 Controlled Variable Selection Conditional modeling setup: Y an outcome of interest (AKA response or dependent variable), X 1,..., X p a set of p potential explanatory variables (AKA covariates, features, or independent variables), Want to learn about the conditional distribution of Y given X 1,..., X p How can we select important explanatory variables with few mistakes? Applications to: Medicine/genetics/health care Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 1 / 15

5 Controlled Variable Selection Conditional modeling setup: Y an outcome of interest (AKA response or dependent variable), X 1,..., X p a set of p potential explanatory variables (AKA covariates, features, or independent variables), Want to learn about the conditional distribution of Y given X 1,..., X p How can we select important explanatory variables with few mistakes? Applications to: Medicine/genetics/health care Economics/political science Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 1 / 15

6 Controlled Variable Selection Conditional modeling setup: Y an outcome of interest (AKA response or dependent variable), X 1,..., X p a set of p potential explanatory variables (AKA covariates, features, or independent variables), Want to learn about the conditional distribution of Y given X 1,..., X p How can we select important explanatory variables with few mistakes? Applications to: Medicine/genetics/health care Economics/political science Industry/technology Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 1 / 15

7 Controlled Variable Selection (cont d) What is an important variable? Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 2 / 15

8 Controlled Variable Selection (cont d) What is an important variable? We consider X j to be unimportant if the conditional distribution of Y given X 1,..., X p does not depend on X j. Formally, X j is unimportant if it is conditionally independent of Y given X -j : Y X j X -j Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 2 / 15

9 Controlled Variable Selection (cont d) What is an important variable? We consider X j to be unimportant if the conditional distribution of Y given X 1,..., X p does not depend on X j. Formally, X j is unimportant if it is conditionally independent of Y given X -j : Y X j X -j Markov Blanket of Y : smallest set S such that Y X -S X S Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 2 / 15

10 Controlled Variable Selection (cont d) What is an important variable? We consider X j to be unimportant if the conditional distribution of Y given X 1,..., X p does not depend on X j. Formally, X j is unimportant if it is conditionally independent of Y given X -j : Y X j X -j Markov Blanket of Y : smallest set S such that Y X -S X S For GLMs with no stochastically redundant covariates, equivalent to {j : β j = 0} Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 2 / 15

11 Controlled Variable Selection (cont d) What is an important variable? We consider X j to be unimportant if the conditional distribution of Y given X 1,..., X p does not depend on X j. Formally, X j is unimportant if it is conditionally independent of Y given X -j : Y X j X -j Markov Blanket of Y : smallest set S such that Y X -S X S For GLMs with no stochastically redundant covariates, equivalent to {j : β j = 0} To make sure we do not make too many mistakes, we seek to select a set Ŝ to control the false discovery rate (FDR): ( ) #{j in FDR(Ŝ) = E Ŝ : X j unimportant} #{j in Ŝ} q (e.g. 10%) Here is a set of variables Ŝ, 90% of which I expect to be important Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 2 / 15

12 How to Incorporate Adaptivity/Selection? Especially important for complex, high-dimensional problems Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 3 / 15

13 How to Incorporate Adaptivity/Selection? Especially important for complex, high-dimensional problems One approach: Post-Selection Inference Idea: give analyst unrestricted access to the data, adjust inference afterwards Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 3 / 15

14 How to Incorporate Adaptivity/Selection? Especially important for complex, high-dimensional problems One approach: Post-Selection Inference Idea: give analyst unrestricted access to the data, adjust inference afterwards Berk et al. (2013) Adjustment does not depend on what the analyst does Generality leads to conservativeness in practice Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 3 / 15

15 How to Incorporate Adaptivity/Selection? Especially important for complex, high-dimensional problems One approach: Post-Selection Inference Idea: give analyst unrestricted access to the data, adjust inference afterwards Berk et al. (2013) Adjustment does not depend on what the analyst does Generality leads to conservativeness in practice Lee et al. (2016) and many others, at/from Stanford and elsewhere Track what analyst does/looks at, condition on it for inference Inference is exact! Not general: adjustment must be specifically Taylored to what analyst looks at Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 3 / 15

16 How to Incorporate Adaptivity/Selection? (cont d) Second approach: slightly restrict analyst s access to data so that inference unaffected by adaptivity/selection Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 4 / 15

17 How to Incorporate Adaptivity/Selection? (cont d) Second approach: slightly restrict analyst s access to data so that inference unaffected by adaptivity/selection Reusable Holdout from differential privacy literature (Dwork et al., 2015) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 4 / 15

18 How to Incorporate Adaptivity/Selection? (cont d) Second approach: slightly restrict analyst s access to data so that inference unaffected by adaptivity/selection Reusable Holdout from differential privacy literature (Dwork et al., 2015) Constrains access to data by adding noise to queries in specific way Subject to that constraint, analyst can arbitrarily use query outcomes for selection Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 4 / 15

19 How to Incorporate Adaptivity/Selection? (cont d) Second approach: slightly restrict analyst s access to data so that inference unaffected by adaptivity/selection Reusable Holdout from differential privacy literature (Dwork et al., 2015) Constrains access to data by adding noise to queries in specific way Subject to that constraint, analyst can arbitrarily use query outcomes for selection Bounds bias of naïve inference, regardless of what analyst does/queries (just depends on number of queries) Generality leads to conservativeness Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 4 / 15

20 How to Incorporate Adaptivity/Selection? (cont d) Second approach: slightly restrict analyst s access to data so that inference unaffected by adaptivity/selection Reusable Holdout from differential privacy literature (Dwork et al., 2015) Constrains access to data by adding noise to queries in specific way Subject to that constraint, analyst can arbitrarily use query outcomes for selection Bounds bias of naïve inference, regardless of what analyst does/queries (just depends on number of queries) Generality leads to conservativeness This talk: Knockoffs Not originally proposed as method for inference after adaptivity/selection Thought about in a different way, fits in well! Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 4 / 15

21 Generic Knockoffs Framework (1) Construct knockoffs: Artificial versions ( knockoffs ) X j of each covariate X j Act as controls for assessing importance of original covariates Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 5 / 15

22 Generic Knockoffs Framework (1) Construct knockoffs: Artificial versions ( knockoffs ) X j of each covariate X j Act as controls for assessing importance of original covariates (2) Compute knockoff statistics: Scalar statistics Z j ( Z j) for each covariate (knockoff) measuring importance Scalar W j combines Z j, Z j to measure relative importance of X j vs. Xj Positive W j denotes original more important, strength measured by magnitude Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 5 / 15

23 Generic Knockoffs Framework (1) Construct knockoffs: Artificial versions ( knockoffs ) X j of each covariate X j Act as controls for assessing importance of original covariates (2) Compute knockoff statistics: Scalar statistics Z j ( Z j) for each covariate (knockoff) measuring importance Scalar W j combines Z j, Z j to measure relative importance of X j vs. Xj Positive W j denotes original more important, strength measured by magnitude (3) Find the knockoff threshold: Order the variables by decreasing W j and proceed down list Select only variables with positive W j until last time negatives positives q Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 5 / 15

24 Generic Knockoffs Framework (1) Construct knockoffs: Artificial versions ( knockoffs ) X j of each covariate X j Act as controls for assessing importance of original covariates (2) Compute knockoff statistics: Scalar statistics Z j ( Z j) for each covariate (knockoff) measuring importance Scalar W j combines Z j, Z j to measure relative importance of X j vs. Xj Positive W j denotes original more important, strength measured by magnitude (3) Find the knockoff threshold: Order the variables by decreasing W j and proceed down list Select only variables with positive W j until last time negatives positives q Coin-flipping property: The key to knockoffs is that steps (1) and (2) are done specifically to ensure that, conditional on W 1,..., W p, the signs of the unimportant/null W j are independently ±1 with probability 1/2 Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 5 / 15

25 Original Knockoffs (Barber and Candès, 2015) Some Notation: y is the n 1 column vector of i.i.d. draws of the random variable Y X j is the n 1 column vector of i.i.d. draws of the random variable X j X j is the knockoff variable (column vector) for X j Design matrices X := [X 1 X p ], X := [ X 1 X p ] Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 6 / 15

26 Original Knockoffs (Barber and Candès, 2015) Some Notation: y is the n 1 column vector of i.i.d. draws of the random variable Y X j is the n 1 column vector of i.i.d. draws of the random variable X j X j is the knockoff variable (column vector) for X j Design matrices X := [X 1 X p ], X := [ X 1 X p ] Original knockoffs procedure: Knockoffs constructed geometrically/deterministically: [ [X X ] [X X X ] = X X X diag{s} X X diag{s} X X ] Finite-sample FDR control and leverages sparsity for power Takes canonical model-based approach: condition on X, assume and rely heavily on low-dimensional (n p) Gaussian linear model Sufficiency requirement: W j function only of [X X ] [X X ] and [X X ] y Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 6 / 15

27 Model-Free Knockoffs (Candès et al., 2016) Shifts the burden of knowledge from Y onto X: Joint distribution of X 1,..., X p can be arbitrary but it must be known, and the observations must be independently and identically disributed (i.i.d.) rows of X = (X i,1,..., X i,p ) iid F Uses F to generate random knockoffs that are swap-exchangeable: [X 1 X j X p X 1 X j X p ] D = [X 1 X j X p X 1 X j X p ] Conditions on y (so no assumptions/model needed for Y X 1,..., X p ) Works for any n and p No sufficiency requirement on W j Robust to overfitting X s distribution Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 7 / 15

28 Model-Free Knockoffs (Candès et al., 2016) Shifts the burden of knowledge from Y onto X: Joint distribution of X 1,..., X p can be arbitrary but it must be known, and the observations must be independently and identically disributed (i.i.d.) rows of X = (X i,1,..., X i,p ) iid F Uses F to generate random knockoffs that are swap-exchangeable: [X 1 X j X p X 1 X j X p ] D = [X 1 X j X p X 1 X j X p ] Conditions on y (so no assumptions/model needed for Y X 1,..., X p ) Works for any n and p No sufficiency requirement on W j Robust to overfitting X s distribution Instead of modeling y and conditioning on X, conditions on y and models X Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 7 / 15

29 Knockoff Statistics (a) Pick a vector-valued function g to compute variable importances Z j, Z j (Z 1,..., Z p, Z 1,..., Z ( p ) := g y, [X X ) ] g must satisfy symmetry property: for any S {1,..., p}, g (y, [X X ) ( ] swap(s) = g y, [X X ) ] swap(s) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 8 / 15

30 Knockoff Statistics (a) Pick a vector-valued function g to compute variable importances Z j, Z j (Z 1,..., Z p, Z 1,..., Z ( p ) := g y, [X X ) ] g must satisfy symmetry property: for any S {1,..., p}, g (y, [X X ) ( ] swap(s) = g y, [X X ) ] swap(s) (b) Pick an antisymmetric function f j : R 2 R, i.e., and set W j = f j (Z j, Z j ) f j (a, b) = f j (b, a) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 8 / 15

31 Original Approach Choose g (with symmetry property) and f j a priori; run entire knockoffs pipeline Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 9 / 15

32 Original Approach Choose g (with symmetry property) and f j a priori; run entire knockoffs pipeline Example: ( g runs lasso y [X X ) ]; λ to get (β 1,..., β p, β 1,..., β p ); takes abs. values f j (a, b) = a b Lasso Coefficient Difference (LCD) statistic: W j = β j β j Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 9 / 15

33 Original Approach Choose g (with symmetry property) and f j a priori; run entire knockoffs pipeline Example: ( g runs lasso y [X X ) ]; λ to get (β 1,..., β p, β 1,..., β p ); takes abs. values f j (a, b) = a b Lasso Coefficient Difference (LCD) statistic: W j = β j β j Allows analyst to use huge range of statistical tools Know little about the data: nonparametric methods like cross-validation, random forests, neural networks Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 9 / 15

34 Original Approach Choose g (with symmetry property) and f j a priori; run entire knockoffs pipeline Example: ( g runs lasso y [X X ) ]; λ to get (β 1,..., β p, β 1,..., β p ); takes abs. values f j (a, b) = a b Lasso Coefficient Difference (LCD) statistic: W j = β j β j Allows analyst to use huge range of statistical tools Know little about the data: nonparametric methods like cross-validation, random forests, neural networks Know more about the data: parametric or Bayesian methods (guarantees do not rely on model/prior being well-specified, nor MCMC converging) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 9 / 15

35 Original Approach Choose g (with symmetry property) and f j a priori; run entire knockoffs pipeline Example: ( g runs lasso y [X X ) ]; λ to get (β 1,..., β p, β 1,..., β p ); takes abs. values f j (a, b) = a b Lasso Coefficient Difference (LCD) statistic: W j = β j β j Allows analyst to use huge range of statistical tools Know little about the data: nonparametric methods like cross-validation, random forests, neural networks Know more about the data: parametric or Bayesian methods (guarantees do not rely on model/prior being well-specified, nor MCMC converging) Cannot use human intuition from looking at data: no adaptivity Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 9 / 15

36 Post-Selection Inference Approach Generate knockoffs (step 1 of knockoffs pipeline); reveal randomly swapped data: y, [X X ] swap(r) where R is uniformly random subset of {1,..., p} (analyst does not know R) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 10 / 15

37 Post-Selection Inference Approach Generate knockoffs (step 1 of knockoffs pipeline); reveal randomly swapped data: y, [X X ] swap(r) where R is uniformly random subset of {1,..., p} (analyst does not know R) Choose antisymmetric f, symmetric g however you want from swapped data No way of distinguishing null X j from X j, so FDR still controlled (exactly!) Potentially a lot of information to distinguish non-null X j from X j, and hence adapt f and g to be more powerful Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 10 / 15

38 Post-Selection Inference Approach Generate knockoffs (step 1 of knockoffs pipeline); reveal randomly swapped data: y, [X X ] swap(r) where R is uniformly random subset of {1,..., p} (analyst does not know R) Choose antisymmetric f, symmetric g however you want from swapped data No way of distinguishing null X j from X j, so FDR still controlled (exactly!) Potentially a lot of information to distinguish non-null X j from X j, and hence adapt f and g to be more powerful Use these data-dependent f and g to compute W (rest of step 2 of knockoffs pipeline) and find the knockoff threshold (step 3) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 10 / 15

39 Exchangeability Endows Coin-Flipping Model-free knockoffs are generated to be swap-exchangeable: for any j, [X X ] swap(j) D = [X X ] Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 11 / 15

40 Exchangeability Endows Coin-Flipping Model-free knockoffs are generated to be swap-exchangeable: for any j, [X X ] swap(j) D = [X X ] Coin-flipping property for W j : for any null covariate j, Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 11 / 15

41 Exchangeability Endows Coin-Flipping Model-free knockoffs are generated to be swap-exchangeable: for any j, [X X ] swap(j) D = [X X ] Coin-flipping property for W j : for any null covariate j, (Z 1,..., Z p, Z 1,..., Z ( p ) swap(j) = g y, [X X ) ] swap(j) symmetry of g = g (y, [X X ) ] swap(j) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 11 / 15

42 Exchangeability Endows Coin-Flipping Model-free knockoffs are generated to be swap-exchangeable: for any j, [X X ] swap(j) D = [X X ] Coin-flipping property for W j : for any null covariate j, (Z 1,..., Z p, Z 1,..., Z ( p ) swap(j) = g y, [X X ) ] swap(j) symmetry of g = g (y, [X X ) ] swap(j) ( D swap-exchangeability, nullity of j, uniformity of R = g y, [X X ) ] = (Z 1,..., Z p, Z 1,..., Z p ) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 11 / 15

43 Exchangeability Endows Coin-Flipping Model-free knockoffs are generated to be swap-exchangeable: for any j, [X X ] swap(j) D = [X X ] Coin-flipping property for W j : for any null covariate j, (Z 1,..., Z p, Z 1,..., Z ( p ) swap(j) = g y, [X X ) ] swap(j) symmetry of g = g (y, [X X ) ] swap(j) ( D swap-exchangeability, nullity of j, uniformity of R = g y, [X X ) ] = (Z 1,..., Z p, Z 1,..., Z p ) W j = f j (Z j, Z j ) D = f j ( Z j, Z j ) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 11 / 15

44 Exchangeability Endows Coin-Flipping Model-free knockoffs are generated to be swap-exchangeable: for any j, [X X ] swap(j) D = [X X ] Coin-flipping property for W j : for any null covariate j, (Z 1,..., Z p, Z 1,..., Z ( p ) swap(j) = g y, [X X ) ] swap(j) symmetry of g = g (y, [X X ) ] swap(j) ( D swap-exchangeability, nullity of j, uniformity of R = g y, [X X ) ] = (Z 1,..., Z p, Z 1,..., Z p ) W j = f j (Z j, Z j ) D = f j ( Z j, Z j ) = f j (Z j, Z j ) = W j Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 11 / 15

45 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

46 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: 0 Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

47 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 10 W 2 W 9 W 7 W 6 0 W 3 W 8 W 1 W 4 W 5 Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

48 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W 6 0 W 8 W 3 W 1 W 4 W 5 Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

49 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W 6 0 W 8 W 3 W 1 W 4 W 5 #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

50 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W 6 0 W 8 W 3 W 1 W 4 W #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

51 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W 6 0 W 8 W 3 W 1 W 4 W #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

52 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W 6 0 W 8 W 3 W 1 W 4 W #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

53 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W 6 0 W 8 W 3 W 1 W 4 W #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

54 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W 6 0 W 8 W 3 W 1 W 4 W #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

55 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W 6 0 W 8 W 3 W 1 W 4 W #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

56 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W 6 0 W 8 W 3 W 1 W 4 W #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

57 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W 6 0 W W 3 W 1 W 4 W #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

58 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W 6 0 W W 3 W 1 W 4 W #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

59 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W W W 3 W 1 W 4 W #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

60 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 9 W 2 W 7 W 10 W W W 3 W 1 W 4 W #{negative W j} #{positive W j} q = 20% Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15 ˆτ

61 Find the Knockoff Threshold Example with p = 10 and q = 20% = 1/5: W 7 W 6 0 W 1 W 4 W 5 S = {1, 4, 5, 6, 7} #{negative W j} #{positive W j} q = 20% ˆτ Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 12 / 15

62 Intuition for FDR Control ( ) #{null Xj selected} FDR = E #{total X j selected} Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 13 / 15

63 Intuition for FDR Control ( ) #{null Xj selected} FDR = E #{total X j selected} ( ) #{null positive Wj > ˆτ} = E #{positive W j > ˆτ} Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 13 / 15

64 Intuition for FDR Control ( ) #{null Xj selected} FDR = E #{total X j selected} ( ) #{null positive Wj > ˆτ} = E #{positive W j > ˆτ} ( ) #{null negative Wj > ˆτ} E #{positive W j > ˆτ} Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 13 / 15

65 Intuition for FDR Control ( ) #{null Xj selected} FDR = E #{total X j selected} ( ) #{null positive Wj > ˆτ} = E #{positive W j > ˆτ} ( ) #{null negative Wj > ˆτ} E #{positive W j > ˆτ} ( ) #{negative Wj > ˆτ} E #{positive W j > ˆτ} q ˆτ Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 13 / 15

66 Power and Robustness Power Non-adaptive statistics (e.g., LCD) already see substantial power gains Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 14 / 15

67 Power and Robustness Power Non-adaptive statistics (e.g., LCD) already see substantial power gains More than doubled power in real GWAS (see also Sesia et al. (2017)) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 14 / 15

68 Power and Robustness Power Non-adaptive statistics (e.g., LCD) already see substantial power gains More than doubled power in real GWAS (see also Sesia et al. (2017)) Advantage of adaptivity hard to measure in simulation Despite generality, post-selection inference knockoffs is not conservative Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 14 / 15

69 Power and Robustness Power Non-adaptive statistics (e.g., LCD) already see substantial power gains More than doubled power in real GWAS (see also Sesia et al. (2017)) Advantage of adaptivity hard to measure in simulation Despite generality, post-selection inference knockoffs is not conservative Robustness Remarkably robust to overfitting X s distribution in-sample Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 14 / 15

70 Power and Robustness Power Non-adaptive statistics (e.g., LCD) already see substantial power gains More than doubled power in real GWAS (see also Sesia et al. (2017)) Advantage of adaptivity hard to measure in simulation Despite generality, post-selection inference knockoffs is not conservative Robustness Remarkably robust to overfitting X s distribution in-sample W j more robust when less sensitive to higher moments of X distribution Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 14 / 15

71 Main Takeaway The knockoffs framework Adaptive: allows unlimited human judgement applied to slightly-masked data Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

72 Main Takeaway The knockoffs framework Adaptive: allows unlimited human judgement applied to slightly-masked data General: wrapper for nearly any measure of variable importance Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

73 Main Takeaway The knockoffs framework Adaptive: allows unlimited human judgement applied to slightly-masked data General: wrapper for nearly any measure of variable importance Powerful: accommodates most powerful tools, yet nonconservative Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

74 Main Takeaway The knockoffs framework Adaptive: allows unlimited human judgement applied to slightly-masked data General: wrapper for nearly any measure of variable importance Powerful: accommodates most powerful tools, yet nonconservative Exact: provably, non-asymptotically, controls FDR Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

75 Appendix Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

76 References Barber, R. F. and Candès, E. J. (2015). Controlling the false discovery rate via knockoffs. Ann. Statist., 43(5): Berk, R., Brown, L., Buja, A., Zhang, K., and Zhao, L. (2013). Valid post-selection inference. Ann. Statist., 41(2): Candès, E., Fan, Y., Janson, L., and Lv, J. (2016). Panning for gold: Model-free knockoffs for high-dimensional controlled variable selection. arxiv preprint arxiv: Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., and Roth, A. (2015). The reusable holdout: Preserving validity in adaptive data analysis. Science, 349(6248): Lee, J. D., Sun, D. L., Sun, Y., and Taylor, J. E. (2016). Exact post-selection inference, with application to the lasso. Ann. Statist., 44(3): Sesia, M., Sabatti, C., and Candès, E. (2017). Gene hunting with knockoffs for hidden markov models. arxiv preprint arxiv: Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

77 Robustness Exact Cov 0.75 Power 0.50 FDR Relative Frobenius Norm Error Relative Frobenius Norm Error Figure: Covariates are AR(1) with autocorrelation coefficient 0.3. n = 800, p = 1500, and target FDR is 10%. Y comes from a binomial linear model with logit link function with 50 nonzero entries. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

78 Robustness Exact Cov Graph. Lasso 0.75 Power 0.50 FDR Relative Frobenius Norm Error Relative Frobenius Norm Error Figure: Covariates are AR(1) with autocorrelation coefficient 0.3. n = 800, p = 1500, and target FDR is 10%. Y comes from a binomial linear model with logit link function with 50 nonzero entries. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

79 Robustness Exact Cov Graph. Lasso 50% Emp. Cov 0.75 Power 0.50 FDR Relative Frobenius Norm Error Relative Frobenius Norm Error Figure: Covariates are AR(1) with autocorrelation coefficient 0.3. n = 800, p = 1500, and target FDR is 10%. Y comes from a binomial linear model with logit link function with 50 nonzero entries. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

80 Robustness Exact Cov Graph. Lasso 50% Emp. Cov 62.5% Emp. Cov 0.75 Power 0.50 FDR Relative Frobenius Norm Error Relative Frobenius Norm Error Figure: Covariates are AR(1) with autocorrelation coefficient 0.3. n = 800, p = 1500, and target FDR is 10%. Y comes from a binomial linear model with logit link function with 50 nonzero entries. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

81 Robustness Exact Cov Graph. Lasso 50% Emp. Cov 62.5% Emp. Cov 0.75 Power % Emp. Cov FDR Relative Frobenius Norm Error Relative Frobenius Norm Error Figure: Covariates are AR(1) with autocorrelation coefficient 0.3. n = 800, p = 1500, and target FDR is 10%. Y comes from a binomial linear model with logit link function with 50 nonzero entries. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

82 Robustness Exact Cov Graph. Lasso 50% Emp. Cov 62.5% Emp. Cov 0.75 Power % Emp. Cov 87.5% Emp. Cov FDR Relative Frobenius Norm Error Relative Frobenius Norm Error Figure: Covariates are AR(1) with autocorrelation coefficient 0.3. n = 800, p = 1500, and target FDR is 10%. Y comes from a binomial linear model with logit link function with 50 nonzero entries. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

83 Robustness Exact Cov Graph. Lasso 50% Emp. Cov 62.5% Emp. Cov 0.75 Power % Emp. Cov 87.5% Emp. Cov FDR % Emp. Cov Relative Frobenius Norm Error Relative Frobenius Norm Error Figure: Covariates are AR(1) with autocorrelation coefficient 0.3. n = 800, p = 1500, and target FDR is 10%. Y comes from a binomial linear model with logit link function with 50 nonzero entries. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

84 Simulations in Low-Dimensional Linear Model Power FDR 0.50 Methods BHq Marginal 0.25 BHq Max Lik. MF Knockoffs Orig. Knockoffs Coefficient Amplitude Coefficient Amplitude Figure: Power and FDR (target is 10%) for MF knockoffs and alternative procedures. The design matrix is i.i.d. N (0, 1/n), n = 3000, p = 1000, and y comes from a Gaussian linear model with 60 nonzero regression coefficients having equal magnitudes and random signs. The noise variance is 1. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

85 Simulations in Low-Dimensional Nonlinear Model Power 0.50 FDR Coefficient Amplitude Methods 0.25 BHq Marginal BHq Max Lik. MF Knockoffs Coefficient Amplitude Figure: Power and FDR (target is 10%) for MF knockoffs and alternative procedures. The design matrix is i.i.d. N (0, 1/n), n = 3000, p = 1000, and y comes from a binomial linear model with logit link function, and 60 nonzero regression coefficients having equal magnitudes and random signs. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

86 Simulations in High Dimensions Power 0.50 FDR Coefficient Amplitude 0.25 Methods BHq Marginal MF Knockoffs Coefficient Amplitude Figure: Power and FDR (target is 10%) for MF knockoffs and alternative procedures. The design matrix is i.i.d. N (0, 1/n), n = 3000, p = 6000, and y comes from a binomial linear model with logit link function, and 60 nonzero regression coefficients having equal magnitudes and random signs. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

87 Simulations in High Dimensions with Dependence Power 0.50 FDR Methods BHq Marginal MF Knockoffs Autocorrelation Coefficient Autocorrelation Coefficient Figure: Power and FDR (target is 10%) for MF knockoffs and alternative procedures. The design matrix has AR(1) columns, and marginally each X j N (0, 1/n). n = 3000, p = 6000, and y follows a binomial linear model with logit link function, and 60 nonzero coefficients with random signs and randomly selected locations. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

88 Genetic Analysis of Crohn s Disease 2007 case-control study by WTCCC n 5, 000, p 375, 000; preprocessing mirrored original analysis Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

89 Genetic Analysis of Crohn s Disease 2007 case-control study by WTCCC n 5, 000, p 375, 000; preprocessing mirrored original analysis Strong spatial structure: second-order knockoffs generated using Approximate SDP on genetic covariance estimate Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

90 Genetic Analysis of Crohn s Disease 2007 case-control study by WTCCC n 5, 000, p 375, 000; preprocessing mirrored original analysis Strong spatial structure: second-order knockoffs generated using Approximate SDP on genetic covariance estimate Entire analysis took 6 hours of serial computation time; 1 hour in parallel Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

91 Genetic Analysis of Crohn s Disease 2007 case-control study by WTCCC n 5, 000, p 375, 000; preprocessing mirrored original analysis Strong spatial structure: second-order knockoffs generated using Approximate SDP on genetic covariance estimate Entire analysis took 6 hours of serial computation time; 1 hour in parallel Model-free knockoffs made twice as many discoveries as original analysis Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

92 Genetic Analysis of Crohn s Disease 2007 case-control study by WTCCC n 5, 000, p 375, 000; preprocessing mirrored original analysis Strong spatial structure: second-order knockoffs generated using Approximate SDP on genetic covariance estimate Entire analysis took 6 hours of serial computation time; 1 hour in parallel Model-free knockoffs made twice as many discoveries as original analysis Some new discoveries confirmed in larger study Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

93 Genetic Analysis of Crohn s Disease 2007 case-control study by WTCCC n 5, 000, p 375, 000; preprocessing mirrored original analysis Strong spatial structure: second-order knockoffs generated using Approximate SDP on genetic covariance estimate Entire analysis took 6 hours of serial computation time; 1 hour in parallel Model-free knockoffs made twice as many discoveries as original analysis Some new discoveries confirmed in larger study Some corroborated by work on nearby genes: promising candidates Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

94 Robustness on Real Data Power 0.50 FDR Coefficient Amplitude Coefficient Amplitude Figure: Power and FDR (target is 10%) for model-free knockoffs applied to subsamples of a chromosome 1 of real genetic design matrix; n 1, 400. Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

95 Careful with Symmetry Property Have to be a bit careful recall symmetry property: for any S {1,..., p} g (y, [X X ) ( ] swap(s) = g y, [X X ) ] e.g., if covariates split into K groups G 1,..., G K swap(s) Incorrect: g runs group lasso with 2K groups G 1,..., G K, G 1,..., G K Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

96 Careful with Symmetry Property Have to be a bit careful recall symmetry property: for any S {1,..., p} g (y, [X X ) ( ] swap(s) = g y, [X X ) ] e.g., if covariates split into K groups G 1,..., G K swap(s) Incorrect: g runs group lasso with 2K groups G 1,..., G K, G 1,..., G K Correct: g runs group lasso with K groups {G 1, G 1 },..., {G K, G K } Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

97 Sequential Independent Pairs Generates Valid Knockoffs Algorithm 1 Sequential Conditional Independent Pairs for j = {1,..., p} do Sample X j from L(X j X -j, X1:j 1 ) conditionally independently of X j end Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

98 Sequential Independent Pairs Generates Valid Knockoffs Algorithm 1 Sequential Conditional Independent Pairs for j = {1,..., p} do Sample X j from L(X j X -j, X1:j 1 ) conditionally independently of X j end Proof sketch (discrete case): Denote PMF of (X 1:p, X 1:j 1 ) by L(X -j, X j, X 1:j 1 ) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

99 Sequential Independent Pairs Generates Valid Knockoffs Algorithm 1 Sequential Conditional Independent Pairs for j = {1,..., p} do Sample X j from L(X j X -j, X1:j 1 ) conditionally independently of X j end Proof sketch (discrete case): Denote PMF of (X 1:p, X 1:j 1 ) by L(X -j, X j, X 1:j 1 ) Conditional PMF of Xj X 1:p, X 1:j 1 is L(X -j, X j, X 1:j 1 ) u L(X -j, u, X 1:j 1 ). Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

100 Sequential Independent Pairs Generates Valid Knockoffs Algorithm 1 Sequential Conditional Independent Pairs for j = {1,..., p} do Sample X j from L(X j X -j, X1:j 1 ) conditionally independently of X j end Proof sketch (discrete case): Denote PMF of (X 1:p, X 1:j 1 ) by L(X -j, X j, X 1:j 1 ) Conditional PMF of Xj X 1:p, X 1:j 1 is L(X -j, X j, X 1:j 1 ) u L(X -j, u, X 1:j 1 ). Joint PMF of (X 1:p, X 1:j ) is L(X -j, X j, X 1:j 1 )L(X -j, X j, X 1:j 1 ) u L(X -j, u, X 1:j 1 ) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

101 Sequential Independent Pairs Generates Valid Knockoffs Algorithm 1 Sequential Conditional Independent Pairs for j = {1,..., p} do Sample X j from L(X j X -j, X1:j 1 ) conditionally independently of X j end Proof sketch (discrete case): Denote PMF of (X 1:p, X 1:j 1 ) by L(X -j, X j, X 1:j 1 ) Conditional PMF of Xj X 1:p, X 1:j 1 is L(X -j, X j, X 1:j 1 ) u L(X -j, u, X 1:j 1 ). Joint PMF of (X 1:p, X 1:j ) is L(X -j, X j, X 1:j 1 )L(X -j, X j, X 1:j 1 ) u L(X -j, u, X 1:j 1 ) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

102 Sequential Independent Pairs Generates Valid Knockoffs Algorithm 1 Sequential Conditional Independent Pairs for j = {1,..., p} do Sample X j from L(X j X -j, X1:j 1 ) conditionally independently of X j end Proof sketch (discrete case): Denote PMF of (X 1:p, X 1:j 1 ) by L(X -j, X j, X 1:j 1 ) Conditional PMF of Xj X 1:p, X 1:j 1 is L(X -j, X j, X 1:j 1 ) u L(X -j, u, X 1:j 1 ). Joint PMF of (X 1:p, X 1:j ) is L(X -j, X j, X 1:j 1 )L(X -j, X j, X 1:j 1 ) u L(X -j, u, X 1:j 1 ) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

103 Sequential Independent Pairs Generates Valid Knockoffs Algorithm 1 Sequential Conditional Independent Pairs for j = {1,..., p} do Sample X j from L(X j X -j, X1:j 1 ) conditionally independently of X j end Proof sketch (discrete case): Denote PMF of (X 1:p, X 1:j 1 ) by L(X -j, X j, X 1:j 1 ) Conditional PMF of Xj X 1:p, X 1:j 1 is L(X -j, X j, X 1:j 1 ) u L(X -j, u, X 1:j 1 ). Joint PMF of (X 1:p, X 1:j ) is L(X -j, X j, X 1:j 1 )L(X -j, X j, X 1:j 1 ) u L(X -j, u, X 1:j 1 ) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

104 Proof of Control ( ) #{null Xj selected} FDR = E #{total X j selected} q ˆτ Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

105 Proof of Control ( ) #{null Xj selected} FDR = E #{total X j selected} ( ) #{null positive Wj > ˆτ} = E #{positive W j > ˆτ} q ˆτ Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

106 Proof of Control ( ) #{null Xj selected} FDR = E #{total X j selected} ( ) #{null positive Wj > ˆτ} = E #{positive W j > ˆτ} ( ) #{null negative Wj > ˆτ} E #{positive W j > ˆτ} q ˆτ Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

107 Proof of Control ( ) #{null Xj selected} FDR = E #{total X j selected} ( ) #{null positive Wj > ˆτ} = E #{positive W j > ˆτ} ( ) #{null negative Wj > ˆτ} E #{positive W j > ˆτ} ( ) #{negative Wj > ˆτ} E #{positive W j > ˆτ} q ˆτ Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

108 Proof of Control ( ) #{null Xj selected} FDR = E #{total X j selected} ( ) #{null positive Wj > ˆτ} = E #{positive W j > ˆτ} ( ) #{null negative Wj > ˆτ} E #{positive W j > ˆτ} ( ) #{negative Wj > ˆτ} E #{positive W j > ˆτ} More precisely: ( ) #{null X mfdr = E j selected} q 1 + #{total X j selected} q ˆτ ( ) #{null positive Wj > ˆτ} = E q 1 + #{positive W j > ˆτ} Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

109 Proof of Control ( ) #{null Xj selected} FDR = E #{total X j selected} ( ) #{null positive Wj > ˆτ} = E #{positive W j > ˆτ} ( ) #{null negative Wj > ˆτ} E #{positive W j > ˆτ} ( ) #{negative Wj > ˆτ} E #{positive W j > ˆτ} More precisely: ( ) ( ) #{null X mfdr = E j selected} #{null positive Wj > ˆτ} q 1 = E + #{total X j selected} q 1 + #{positive W j > ˆτ} ( #{null positive Wj > ˆτ} = E 1 + #{null negative W j > ˆτ} 1 + #{null negative W ) j > ˆτ} q 1 + #{positive W j > ˆτ} q ˆτ Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

110 Proof of Control ( ) #{null Xj selected} FDR = E #{total X j selected} ( ) #{null positive Wj > ˆτ} = E #{positive W j > ˆτ} ( ) #{null negative Wj > ˆτ} E #{positive W j > ˆτ} ( ) #{negative Wj > ˆτ} E #{positive W j > ˆτ} More precisely: ( ) #{null X mfdr = E j selected} q 1 + #{total X j selected} = E q ˆτ ( #{null positive Wj > ˆτ} = E q 1 + #{positive W j > ˆτ} ( #{null positive Wj > ˆτ} 1 + #{null negative W j > ˆτ} 1 + #{null negative W j > ˆτ} q 1 + #{positive W j > ˆτ} }{{} q by definition of ˆτ ) ) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15

111 Proof of Control ( ) #{null Xj selected} FDR = E #{total X j selected} ( ) #{null positive Wj > ˆτ} = E #{positive W j > ˆτ} ( ) #{null negative Wj > ˆτ} E #{positive W j > ˆτ} ( ) #{negative Wj > ˆτ} E #{positive W j > ˆτ} More precisely: ( ) #{null X mfdr = E j selected} q 1 + #{total X j selected} ( #{null positive Wj > ˆτ} = E 1 + #{null negative W j > ˆτ} } {{ } Supermartingale 1 with ˆτ a stopping time q ˆτ ( #{null positive Wj > ˆτ} = E q 1 + #{positive W j > ˆτ} 1 + #{null negative W j > ˆτ} q 1 + #{positive W j > ˆτ} }{{} q by definition of ˆτ ) ) Lucas Janson (Harvard Statistics) Knockoffs as Post-Selection Inference 15 / 15