Counterfactual Reasoning in Algorithmic Fairness

Transcription

1 Counterfactual Reasoning in Algorithmic Fairness Ricardo Silva University College London and The Alan Turing Institute Joint work with Matt Kusner (Warwick/Turing), Chris Russell (Sussex/Turing), and Joshua Loftus (NYU)

2 Fairness and Machine Learning The dream: if we teach machines to perform sensitive decisions, they will not suffer from human biases. The reality: the GIGO principle still holds, regardless of whether we are talking of statistical models or software.

3 The Message There is only so much data alone can tell you about fairness. I m not talking about just value judgments. We should highlight the role that the datagenerating causal process has in shaping our notions of fairness.

4 Nobody is Saying This is Easy At no point I will suggest that building a causal model is easy. Some untested and untestable assumptions will be needed. The idea is to make your assumptions as explicit as possible, hopefully being less wrong in the end.

5 <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixv1l86vq1rn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlt1eldvesarh6kc0vo5wtdmajnchizrxwyasblykjn1yqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha1_base64="h0vkymv1txiryc/6suot3b8szzo=">aaaczxicdzfnb9naeibx5queaoerlxuviaokbc4gtlrcobajtie4isbrcbzkfli746arzykdcph3uper+aus0wg1bzntq5nzd2eeywslpsxj7yi+cfpw7tt7dwf39u8/edh8th/sbemejovv1k1y8kikwtfjujiphylofz7ky/d9/equnzfwfkj1jtmncynlkybcaj78m3nln/om8iycbnmguq7aeoeah757ya3jqork3hb6bksohkttyfnznkc6lbsc7hhqhvut2llcqvhcmoiudfj8hwcct0akq6wqil5cvrknkriss+rkedizkij6h6xv2lujgp4nsukl63bdyd48sebjjvhvkw7fadvg0xz4kyusadqaegq8n6zjtbmwhemhsbtkjccaxbiwoa3sgey/azeh6fizkcl4gscw1hdfzc++aef7v9z56nralb9c65px1aynlw9mrtr1ogre+udlo3os/zv5iv1gr9zbghayzmpfbq7cpzzviasxv74qjl+n0msufkzyhnvcnrixlgwv2tv2gr2xmrprogqjb9h3+ev8nf5xjiuottwes52if/4farpjeg==</latexit> <latexit sha1_base64="h0vkymv1txiryc/6suot3b8szzo=">aaaczxicdzfnb9naeibx5queaoerlxuviaokbc4gtlrcobajtie4isbrcbzkfli746arzykdcph3uper+aus0wg1bzntq5nzd2eeywslpsxj7yi+cfpw7tt7dwf39u8/edh8th/sbemejovv1k1y8kikwtfjujiphylofz7ky/d9/equnzfwfkj1jtmncynlkybcaj78m3nln/om8iycbnmguq7aeoeah757ya3jqork3hb6bksohkttyfnznkc6lbsc7hhqhvut2llcqvhcmoiudfj8hwcct0akq6wqil5cvrknkriss+rkedizkij6h6xv2lujgp4nsukl63bdyd48sebjjvhvkw7fadvg0xz4kyusadqaegq8n6zjtbmwhemhsbtkjccaxbiwoa3sgey/azeh6fizkcl4gscw1hdfzc++aef7v9z56nralb9c65px1aynlw9mrtr1ogre+udlo3os/zv5iv1gr9zbghayzmpfbq7cpzzviasxv74qjl+n0msufkzyhnvcnrixlgwv2tv2gr2xmrprogqjb9h3+ev8nf5xjiuottwes52if/4farpjeg==</latexit> <latexit sha1_base64="jgc96zbtujoycxypdl+iax6pzeu=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhk9glitmcf4ylr3ujtvrnn0lh17miebkmisbxaics/jru/gr+a043qdnez09ob5+ezn2mlpkmo+h2ev65eu35j7+bg1u07d+8n7z84cqa2aifckgonkthuuuoejcmcvhahtbuep6u3xf/4bk2trn+gtyxzepza5liaewox/dn9xz/yhpaz2blhczzv0eqivft07xnulacik9oa0hgtc9bc6kyeykwg1sbj4pwoarxym1twearc7ixbe6ndx/84englkh9ixqdxnfk1qvxglvwhj8p9npku1dkknwdxxgf0b5ctzq/btopfcbsno23xiyduwyj1dbgy/koyi+osnqkfzs3iqkj5a5akungoktphbwifs5x5qkfen2+2h2n5e89kppch5kyt37jnxzrqorcpu68sgqp3vterl/vmneuv543ulu9vi9op8lp1sxrx5pm0pnu18qcelx5wlgqw4o9hxrdcfh7li+bofxxh4/h9ndp408exxx6xx+wzi9kldsdesum2yskybe3wnfgwfgq/hn/dh6fsmojfpgq7ff78c0xy5lo=</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixv1l86vq1rn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlt1eldvesarh6kc0vo5wtdmajnchizrxwyasblykjn1yqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha1_base64="h0vkymv1txiryc/6suot3b8szzo=">aaaczxicdzfnb9naeibx5queaoerlxuviaokbc4gtlrcobajtie4isbrcbzkfli746arzykdcph3uper+aus0wg1bzntq5nzd2eeywslpsxj7yi+cfpw7tt7dwf39u8/edh8th/sbemejovv1k1y8kikwtfjujiphylofz7ky/d9/equnzfwfkj1jtmncynlkybcaj78m3nln/om8iycbnmguq7aeoeah757ya3jqork3hb6bksohkttyfnznkc6lbsc7hhqhvut2llcqvhcmoiudfj8hwcct0akq6wqil5cvrknkriss+rkedizkij6h6xv2lujgp4nsukl63bdyd48sebjjvhvkw7fadvg0xz4kyusadqaegq8n6zjtbmwhemhsbtkjccaxbiwoa3sgey/azeh6fizkcl4gscw1hdfzc++aef7v9z56nralb9c65px1aynlw9mrtr1ogre+udlo3os/zv5iv1gr9zbghayzmpfbq7cpzzviasxv74qjl+n0msufkzyhnvcnrixlgwv2tv2gr2xmrprogqjb9h3+ev8nf5xjiuottwes52if/4farpjeg==</latexit> <latexit sha1_base64="h0vkymv1txiryc/6suot3b8szzo=">aaaczxicdzfnb9naeibx5queaoerlxuviaokbc4gtlrcobajtie4isbrcbzkfli746arzykdcph3uper+aus0wg1bzntq5nzd2eeywslpsxj7yi+cfpw7tt7dwf39u8/edh8th/sbemejovv1k1y8kikwtfjujiphylofz7ky/d9/equnzfwfkj1jtmncynlkybcaj78m3nln/om8iycbnmguq7aeoeah757ya3jqork3hb6bksohkttyfnznkc6lbsc7hhqhvut2llcqvhcmoiudfj8hwcct0akq6wqil5cvrknkriss+rkedizkij6h6xv2lujgp4nsukl63bdyd48sebjjvhvkw7fadvg0xz4kyusadqaegq8n6zjtbmwhemhsbtkjccaxbiwoa3sgey/azeh6fizkcl4gscw1hdfzc++aef7v9z56nralb9c65px1aynlw9mrtr1ogre+udlo3os/zv5iv1gr9zbghayzmpfbq7cpzzviasxv74qjl+n0msufkzyhnvcnrixlgwv2tv2gr2xmrprogqjb9h3+ev8nf5xjiuottwes52if/4farpjeg==</latexit> <latexit sha1_base64="jgc96zbtujoycxypdl+iax6pzeu=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhk9glitmcf4ylr3ujtvrnn0lh17miebkmisbxaics/jru/gr+a043qdnez09ob5+ezn2mlpkmo+h2ev65eu35j7+bg1u07d+8n7z84cqa2aifckgonkthuuuoejcmcvhahtbuep6u3xf/4bk2trn+gtyxzepza5liaewox/dn9xz/yhpaz2blhczzv0eqivft07xnulacik9oa0hgtc9bc6kyeykwg1sbj4pwoarxym1twearc7ixbe6ndx/84englkh9ixqdxnfk1qvxglvwhj8p9npku1dkknwdxxgf0b5ctzq/btopfcbsno23xiyduwyj1dbgy/koyi+osnqkfzs3iqkj5a5akungoktphbwifs5x5qkfen2+2h2n5e89kppch5kyt37jnxzrqorcpu68sgqp3vterl/vmneuv543ulu9vi9op8lp1sxrx5pm0pnu18qcelx5wlgqw4o9hxrdcfh7li+bofxxh4/h9ndp408exxx6xx+wzi9kldsdesum2yskybe3wnfgwfgq/hn/dh6fsmojfpgq7ff78c0xy5lo=</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixv1l86vq1rn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlt1eldvesarh6kc0vo5wtdmajnchizrxwyasblykjn1yqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha1_base64="h0vkymv1txiryc/6suot3b8szzo=">aaaczxicdzfnb9naeibx5queaoerlxuviaokbc4gtlrcobajtie4isbrcbzkfli746arzykdcph3uper+aus0wg1bzntq5nzd2eeywslpsxj7yi+cfpw7tt7dwf39u8/edh8th/sbemejovv1k1y8kikwtfjujiphylofz7ky/d9/equnzfwfkj1jtmncynlkybcaj78m3nln/om8iycbnmguq7aeoeah757ya3jqork3hb6bksohkttyfnznkc6lbsc7hhqhvut2llcqvhcmoiudfj8hwcct0akq6wqil5cvrknkriss+rkedizkij6h6xv2lujgp4nsukl63bdyd48sebjjvhvkw7fadvg0xz4kyusadqaegq8n6zjtbmwhemhsbtkjccaxbiwoa3sgey/azeh6fizkcl4gscw1hdfzc++aef7v9z56nralb9c65px1aynlw9mrtr1ogre+udlo3os/zv5iv1gr9zbghayzmpfbq7cpzzviasxv74qjl+n0msufkzyhnvcnrixlgwv2tv2gr2xmrprogqjb9h3+ev8nf5xjiuottwes52if/4farpjeg==</latexit> <latexit sha1_base64="h0vkymv1txiryc/6suot3b8szzo=">aaaczxicdzfnb9naeibx5queaoerlxuviaokbc4gtlrcobajtie4isbrcbzkfli746arzykdcph3uper+aus0wg1bzntq5nzd2eeywslpsxj7yi+cfpw7tt7dwf39u8/edh8th/sbemejovv1k1y8kikwtfjujiphylofz7ky/d9/equnzfwfkj1jtmncynlkybcaj78m3nln/om8iycbnmguq7aeoeah757ya3jqork3hb6bksohkttyfnznkc6lbsc7hhqhvut2llcqvhcmoiudfj8hwcct0akq6wqil5cvrknkriss+rkedizkij6h6xv2lujgp4nsukl63bdyd48sebjjvhvkw7fadvg0xz4kyusadqaegq8n6zjtbmwhemhsbtkjccaxbiwoa3sgey/azeh6fizkcl4gscw1hdfzc++aef7v9z56nralb9c65px1aynlw9mrtr1ogre+udlo3os/zv5iv1gr9zbghayzmpfbq7cpzzviasxv74qjl+n0msufkzyhnvcnrixlgwv2tv2gr2xmrprogqjb9h3+ev8nf5xjiuottwes52if/4farpjeg==</latexit> <latexit sha1_base64="jgc96zbtujoycxypdl+iax6pzeu=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhk9glitmcf4ylr3ujtvrnn0lh17miebkmisbxaics/jru/gr+a043qdnez09ob5+ezn2mlpkmo+h2ev65eu35j7+bg1u07d+8n7z84cqa2aifckgonkthuuuoejcmcvhahtbuep6u3xf/4bk2trn+gtyxzepza5liaewox/dn9xz/yhpaz2blhczzv0eqivft07xnulacik9oa0hgtc9bc6kyeykwg1sbj4pwoarxym1twearc7ixbe6ndx/84englkh9ixqdxnfk1qvxglvwhj8p9npku1dkknwdxxgf0b5ctzq/btopfcbsno23xiyduwyj1dbgy/koyi+osnqkfzs3iqkj5a5akungoktphbwifs5x5qkfen2+2h2n5e89kppch5kyt37jnxzrqorcpu68sgqp3vterl/vmneuv543ulu9vi9op8lp1sxrx5pm0pnu18qcelx5wlgqw4o9hxrdcfh7li+bofxxh4/h9ndp408exxx6xx+wzi9kldsdesum2yskybe3wnfgwfgq/hn/dh6fsmojfpgq7ff78c0xy5lo=</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixv1l86vq1rn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlt1eldvesarh6kc0vo5wtdmajnchizrxwyasblykjn1yqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha1_base64="h0vkymv1txiryc/6suot3b8szzo=">aaaczxicdzfnb9naeibx5queaoerlxuviaokbc4gtlrcobajtie4isbrcbzkfli746arzykdcph3uper+aus0wg1bzntq5nzd2eeywslpsxj7yi+cfpw7tt7dwf39u8/edh8th/sbemejovv1k1y8kikwtfjujiphylofz7ky/d9/equnzfwfkj1jtmncynlkybcaj78m3nln/om8iycbnmguq7aeoeah757ya3jqork3hb6bksohkttyfnznkc6lbsc7hhqhvut2llcqvhcmoiudfj8hwcct0akq6wqil5cvrknkriss+rkedizkij6h6xv2lujgp4nsukl63bdyd48sebjjvhvkw7fadvg0xz4kyusadqaegq8n6zjtbmwhemhsbtkjccaxbiwoa3sgey/azeh6fizkcl4gscw1hdfzc++aef7v9z56nralb9c65px1aynlw9mrtr1ogre+udlo3os/zv5iv1gr9zbghayzmpfbq7cpzzviasxv74qjl+n0msufkzyhnvcnrixlgwv2tv2gr2xmrprogqjb9h3+ev8nf5xjiuottwes52if/4farpjeg==</latexit> <latexit sha1_base64="h0vkymv1txiryc/6suot3b8szzo=">aaaczxicdzfnb9naeibx5queaoerlxuviaokbc4gtlrcobajtie4isbrcbzkfli746arzykdcph3uper+aus0wg1bzntq5nzd2eeywslpsxj7yi+cfpw7tt7dwf39u8/edh8th/sbemejovv1k1y8kikwtfjujiphylofz7ky/d9/equnzfwfkj1jtmncynlkybcaj78m3nln/om8iycbnmguq7aeoeah757ya3jqork3hb6bksohkttyfnznkc6lbsc7hhqhvut2llcqvhcmoiudfj8hwcct0akq6wqil5cvrknkriss+rkedizkij6h6xv2lujgp4nsukl63bdyd48sebjjvhvkw7fadvg0xz4kyusadqaegq8n6zjtbmwhemhsbtkjccaxbiwoa3sgey/azeh6fizkcl4gscw1hdfzc++aef7v9z56nralb9c65px1aynlw9mrtr1ogre+udlo3os/zv5iv1gr9zbghayzmpfbq7cpzzviasxv74qjl+n0msufkzyhnvcnrixlgwv2tv2gr2xmrprogqjb9h3+ev8nf5xjiuottwes52if/4farpjeg==</latexit> <latexit sha1_base64="jgc96zbtujoycxypdl+iax6pzeu=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhk9glitmcf4ylr3ujtvrnn0lh17miebkmisbxaics/jru/gr+a043qdnez09ob5+ezn2mlpkmo+h2ev65eu35j7+bg1u07d+8n7z84cqa2aifckgonkthuuuoejcmcvhahtbuep6u3xf/4bk2trn+gtyxzepza5liaewox/dn9xz/yhpaz2blhczzv0eqivft07xnulacik9oa0hgtc9bc6kyeykwg1sbj4pwoarxym1twearc7ixbe6ndx/84englkh9ixqdxnfk1qvxglvwhj8p9npku1dkknwdxxgf0b5ctzq/btopfcbsno23xiyduwyj1dbgy/koyi+osnqkfzs3iqkj5a5akungoktphbwifs5x5qkfen2+2h2n5e89kppch5kyt37jnxzrqorcpu68sgqp3vterl/vmneuv543ulu9vi9op8lp1sxrx5pm0pnu18qcelx5wlgqw4o9hxrdcfh7li+bofxxh4/h9ndp408exxx6xx+wzi9kldsdesum2yskybe3wnfgwfgq/hn/dh6fsmojfpgq7ff78c0xy5lo=</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixv1l86vq1rn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlt1eldvesarh6kc0vo5wtdmajnchizrxwyasblykjn1yqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha1_base64="h0vkymv1txiryc/6suot3b8szzo=">aaaczxicdzfnb9naeibx5queaoerlxuviaokbc4gtlrcobajtie4isbrcbzkfli746arzykdcph3uper+aus0wg1bzntq5nzd2eeywslpsxj7yi+cfpw7tt7dwf39u8/edh8th/sbemejovv1k1y8kikwtfjujiphylofz7ky/d9/equnzfwfkj1jtmncynlkybcaj78m3nln/om8iycbnmguq7aeoeah757ya3jqork3hb6bksohkttyfnznkc6lbsc7hhqhvut2llcqvhcmoiudfj8hwcct0akq6wqil5cvrknkriss+rkedizkij6h6xv2lujgp4nsukl63bdyd48sebjjvhvkw7fadvg0xz4kyusadqaegq8n6zjtbmwhemhsbtkjccaxbiwoa3sgey/azeh6fizkcl4gscw1hdfzc++aef7v9z56nralb9c65px1aynlw9mrtr1ogre+udlo3os/zv5iv1gr9zbghayzmpfbq7cpzzviasxv74qjl+n0msufkzyhnvcnrixlgwv2tv2gr2xmrprogqjb9h3+ev8nf5xjiuottwes52if/4farpjeg==</latexit> <latexit sha1_base64="h0vkymv1txiryc/6suot3b8szzo=">aaaczxicdzfnb9naeibx5queaoerlxuviaokbc4gtlrcobajtie4isbrcbzkfli746arzykdcph3uper+aus0wg1bzntq5nzd2eeywslpsxj7yi+cfpw7tt7dwf39u8/edh8th/sbemejovv1k1y8kikwtfjujiphylofz7ky/d9/equnzfwfkj1jtmncynlkybcaj78m3nln/om8iycbnmguq7aeoeah757ya3jqork3hb6bksohkttyfnznkc6lbsc7hhqhvut2llcqvhcmoiudfj8hwcct0akq6wqil5cvrknkriss+rkedizkij6h6xv2lujgp4nsukl63bdyd48sebjjvhvkw7fadvg0xz4kyusadqaegq8n6zjtbmwhemhsbtkjccaxbiwoa3sgey/azeh6fizkcl4gscw1hdfzc++aef7v9z56nralb9c65px1aynlw9mrtr1ogre+udlo3os/zv5iv1gr9zbghayzmpfbq7cpzzviasxv74qjl+n0msufkzyhnvcnrixlgwv2tv2gr2xmrprogqjb9h3+ev8nf5xjiuottwes52if/4farpjeg==</latexit> <latexit sha1_base64="jgc96zbtujoycxypdl+iax6pzeu=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhk9glitmcf4ylr3ujtvrnn0lh17miebkmisbxaics/jru/gr+a043qdnez09ob5+ezn2mlpkmo+h2ev65eu35j7+bg1u07d+8n7z84cqa2aifckgonkthuuuoejcmcvhahtbuep6u3xf/4bk2trn+gtyxzepza5liaewox/dn9xz/yhpaz2blhczzv0eqivft07xnulacik9oa0hgtc9bc6kyeykwg1sbj4pwoarxym1twearc7ixbe6ndx/84englkh9ixqdxnfk1qvxglvwhj8p9npku1dkknwdxxgf0b5ctzq/btopfcbsno23xiyduwyj1dbgy/koyi+osnqkfzs3iqkj5a5akungoktphbwifs5x5qkfen2+2h2n5e89kppch5kyt37jnxzrqorcpu68sgqp3vterl/vmneuv543ulu9vi9op8lp1sxrx5pm0pnu18qcelx5wlgqw4o9hxrdcfh7li+bofxxh4/h9ndp408exxx6xx+wzi9kldsdesum2yskybe3wnfgwfgq/hn/dh6fsmojfpgq7ff78c0xy5lo=</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> <latexit sha1_base64="gz8rxw10rufhocup5exuumqzf3s=">aaac2hicdvfnj9mwehxc11k+chy5wfqgdqhkebki0wixjotedwtnvu2cswpvssn7sqwkineaia78ng78cv4ctjdc291ltk9vnp9n3qsvko6i6hcqxrp85eq1veudgzdv3b4zvhvv0jnacpwio4ydpubqsy0tkqrwwlmemlv4lk7edv2jy7rogv2bnhxos1hqmusb5knf8m/0fx/me8lpzmugj1hwrzmjug3rtu+5sryirexrqsdnzkfzqtn5llmavjskg9c7blrgb1jzqygis1mg7yuoh//jqgcxsh6idqhe18jxplyzv3kfnaz3e2zsuoeqdijdgx/qv0g2or9u2w4ww1e0jrbfz4o4bypw18fi+cvjjkhl1cquodelo4rmdvisqme7sgqhfygvlhhmoyys3bzzhqbljzyt8dyhmbtnfmueftfa6dymtl2ybcrc2v5hxtsb1zs/nddsvz5vlu4+ymvvxdjdmwfs+uzvxgmqvvpzusjagr+hdv0i8dmvz4pdz+m4gsfvn4/23/rx7leh7cf7wml2gu2zd+yatzgijketfa2+hz/cl+h38mejnaz6n/fztou//wjnmos+</latexit> The Scope of this Talk We consider prediction and intervention problems (more of the former). In prediction problems, we will have: X : features, or attributes of an individual A :theprotected attributes of an individual Y : thetarget, what we would like to predict Ŷ : our prediction

6 Prediction Problems Prediction here means inferring a property Y that will be used for decision making. For example: Y = 1 means this person will default on a loan (for the decision, should I give this person a loan?) Y = 1 means this person will commit a crime in two years (for the decision, should I release this convict now? ) We would like to predict Y in a fair way, meaning that our predictions should not be biased against particular instances of A.

7 Ŷ<latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> Primitives Even if we take the choice of what goes in A as a primitive, it is still not obvious what we mean by being fair. A first idea: ensure that does not use A. This is known to be unsatisfactory.

8 Ŷ<latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> Ŷ<latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> Ŷ<latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> <latexit sha1_base64="m+zgxsf9wvacj2hahalwrb9eawm=">aaab7xicbvdlsgnbeoz1gemr6thlyba8hv0r9bj04jgceuiyhnnjbdjmdmaz6rvcyd948aciv//hm3/jjnmdjhy0ffxddhdfqrqwff/bw1ldw9/ylgwvt3d29/zlb4cnqzpdej1pqu0ropzloxgdburesg2nssr5mxretp3mezdwahwpo5shce0reqtg0umnzoaieeiwyn7fn4eskyanzchr65a+oj3nsoqrzjja2w78fmmxnsiy5jnij7m8pwxi+7ztqkijt+f4du2endqlr2jtxckkm/x3xjgm1o6syhumfad20zuk/3ntdoorccxumifxbl4ozirbtaavk54wnkecoukzee5wwgbuuiyuokilivh8ezk0ziubxwnulsrv6zyoahzdczxbajdqhvuoqr0ypmizvmkbp70x7937mleuepnmefyb9/kdefyoya==</latexit> Examples Equalized odds: given the outcome Y, attribute A provides no further information about my prediction. Calibration: given my prediction, attribute A provides no further information about the outcome Y. If A is on average informative of Y, we cannot reconcile the above. Remember, here we do not control Y (directly). We decide on predictor.

9 Putting It in the Context of a Causal Model A toy model: imagine A is race, X is owns a red car and Y is crashes car in one year. Let s (informally) draw a causal diagram showing cause-effect relationships among those. It will include a unobserved trait U measuring aggressiveness. We will get into more formal definitions later.

10 A Causal Diagram

11 Some Initial Conclusions A is not a cause of Y. If we build a predictor based on X, it tells us something both about A and about U. Hence, our predictor will be different for different values of A, which does not seem appropriate.

12 A Second Causal Diagram

13 Which Conclusions? A is now a cause of Y (indirectly). It is now impossible to satisfy both equalized odds and calibration simultaneously. Judgment call: is the pathway A è X èy fair?

14 Zooming In, with Another Example A here stands for race, Y for loan default. Same idea, augmented with a mediator:

15 A Causal Primitive: Counterfactual Fairness If we have some protected attribute like race, and a decision such as length of sentence, then our decision satisfies counterfactual fairness if had the protected attributes (e.g., race) of the individual been different, other things being equal, the decision would have remained the same A causal model is necessary to infer such claims from data.

16 Workflow Regardless of the machine learning algorithm to be used, work with a domain expert to estimate a causal model of your data. It s a model of the world, not of your software. Choose any machine learning algorithm of interest, any black-box that takes as inputs observed and unobserved variables in your domain. Select a set of variables based on which sets respect counterfactual fairness. If necessary, infer unobserved variables from the observed ones.

17 Formalizing the Idea Formal notions of counterfactuals date back at least to Jerzy Neyman in the 1920s. I will follow mostly the Structural Causal Model (SCM) framework of Judea Pearl, which has close links to the work of James Robins, and that of Spirtes, Glymour and Scheines.

18 Structural Causal Models A directed acyclic graph (DAG) postulates direct cause-effect pairs. Each vertex in the graph is a random variable in a distribution. Each variable V is given an equation that deterministically defines the value of V as a function of its parents. Such equations are postulated to be structural, in the sense that it follows the cause-effect direction.

19 The Operational Meaning This DAG with equations is causal in the sense that it must encode the effects of a perfect intervention. U Rain U Barometer Rain = f R (U Rain ) Barometer = f B (Rain, U Barometer ) Rain Barometer

20 Interventions Another primitive. It is a overriding operator, sets a variable to a fixed value of interest. Lower case here represents constants. U Rain U Barometer Rain = r Barometer = f B (r, U Barometer ) r Barometer

21 Interventions It is the notion of intervention that leads to the asymmetric nature of causality. U Rain U Barometer Rain = f R (U Rain ) Barometer = b Rain b

22 Interventions It is the notion of intervention that explains why correlation is not causation. Surviving old age Surviving old age Moving to Florida Dying of old age yes Dying of old age

23 Notation: the do Operator We must express how Dying of old age varies with Moving to Florida in both cases. Traditionally, conditional probabilities can be used for that. But notice that, in our example, P(Dying of old age = True Moving to Florida = True) P(Dying of old age = True Moving to Florida = False) is true in the observational case (no intervention), but false in the interventional case.

24 Notation: the do Operator In Pearl s calculus, this is distinguished by using the do operator to indicate an intervention as opposed to an observation. P(Dying of old age = True Moving to Florida = True) P(Dying of old age = True Moving to Florida = False) P(Dying of old age = True do(moving to Florida = True)) = P(Dying of old age = True do(moving to Florida = False))

25 Averages Vs. Individuals This type of notation can be used to express whether a drug is effective or not, averaging over a population, using a randomized controlled trial: P(Healthy = True do(treatment = Drug))?= P(Healthy = True do(treatment = Placebo)) It does not make any claims, however, on whether there is a balance of positive/ negative cases that cancel out.

26 Notation: Counterfactual Indices Meant to capture individual-level variability. For V j a variable in the system, and V i a variable being intervened at value v, we use V j (v) as the counterfactual value of V j, had V i being set to v. Context will tell us which variable the value v refers to.

27 Example Notice: it is common to represent V j (v) as just V j if V i is not a (direct or indirect) cause of V j. U rain (r) U barometer (r) U rain U barometer r Barometer(r) r Barometer(r)

28 Other Things Being Equal That is, A counterfactual value replaces the cause of interest The counterfactual value propagates downstream the causal graph via the structural equations Everything else remains the same ( other things being equal ), i.e., the non-descendants of the manipulated variable.

29 Multiple Worlds A counterfactual is just a different version of the same individual. All versions co-exist in one big joint distribution. U Rain U Barometer Rain Barometer r Barometer(r) r Barometer(r )

30 Workflow Regardless of the machine learning algorithm to be used, work with a domain expert to estimate a causal model of your data. It s a model of the world, not of your software. Choose any machine learning algorithm of interest, any black-box that takes as inputs observed and unobserved variables in your domain. Select a set of variables based on which sets respect counterfactual fairness. If necessary, infer unobserved variables from the observed ones.

31 Back to Counterfactual Fairness The law of counterfactual propagation means that, if we want to ensure P (Ŷ (a) =y A = a, X = x) =P (Ŷ (a0 )=y A = a, X = x) <latexit sha1_base64="ika4uwrxxdg1jbktp7zsue5mo/g=">aaacl3icdvdlsgmxfm34rpu16tjnsigvpmyiobuhkojlcvyhnahcsdm2npmgyyhd7r+58ve6evherx9hpi2orr4iozxzlsk9xsszvjb1yszmzs0vlgawsssrq2vr5szmryaxilrmqh6kmgeschbqsmkk01okkpgep1wve5h61tsqjaudg5ve1pwhhbawi6c01davs3mnawrf5mefn+lewq8optmmdnbnx/ep+p3z+yfumhnwwroctxn7thjojfldhdjnkmq+drthigxdtill9kaorjjtz51y0ghif9q0rmkappvub7hvh+9qpylbodanuhio/pzogs9l4ns66ypqyekvff/y6rfqnbg9fksxogezpdskovyhtsvdtsyoutzrbihg+q+ydeaaubrirc7bnlx5mlqoc7zvsk+pcsxzcr0zti12ub7z6bgv0ruqotii6ben0ct6m56mz+pd+bhfz4zxzbb6bepzc/vyork=</latexit> <latexit sha1_base64="ika4uwrxxdg1jbktp7zsue5mo/g=">aaacl3icdvdlsgmxfm34rpu16tjnsigvpmyiobuhkojlcvyhnahcsdm2npmgyyhd7r+58ve6evherx9hpi2orr4iozxzlsk9xsszvjb1yszmzs0vlgawsssrq2vr5szmryaxilrmqh6kmgeschbqsmkk01okkpgep1wve5h61tsqjaudg5ve1pwhhbawi6c01davs3mnawrf5mefn+lewq8optmmdnbnx/ep+p3z+yfumhnwwroctxn7thjojfldhdjnkmq+drthigxdtill9kaorjjtz51y0ghif9q0rmkappvub7hvh+9qpylbodanuhio/pzogs9l4ns66ypqyekvff/y6rfqnbg9fksxogezpdskovyhtsvdtsyoutzrbihg+q+ydeaaubrirc7bnlx5mlqoc7zvsk+pcsxzcr0zti12ub7z6bgv0ruqotii6ben0ct6m56mz+pd+bhfz4zxzbb6bepzc/vyork=</latexit> <latexit sha1_base64="ika4uwrxxdg1jbktp7zsue5mo/g=">aaacl3icdvdlsgmxfm34rpu16tjnsigvpmyiobuhkojlcvyhnahcsdm2npmgyyhd7r+58ve6evherx9hpi2orr4iozxzlsk9xsszvjb1yszmzs0vlgawsssrq2vr5szmryaxilrmqh6kmgeschbqsmkk01okkpgep1wve5h61tsqjaudg5ve1pwhhbawi6c01davs3mnawrf5mefn+lewq8optmmdnbnx/ep+p3z+yfumhnwwroctxn7thjojfldhdjnkmq+drthigxdtill9kaorjjtz51y0ghif9q0rmkappvub7hvh+9qpylbodanuhio/pzogs9l4ns66ypqyekvff/y6rfqnbg9fksxogezpdskovyhtsvdtsyoutzrbihg+q+ydeaaubrirc7bnlx5mlqoc7zvsk+pcsxzcr0zti12ub7z6bgv0ruqotii6ben0ct6m56mz+pd+bhfz4zxzbb6bepzc/vyork=</latexit> <latexit sha1_base64="ika4uwrxxdg1jbktp7zsue5mo/g=">aaacl3icdvdlsgmxfm34rpu16tjnsigvpmyiobuhkojlcvyhnahcsdm2npmgyyhd7r+58ve6evherx9hpi2orr4iozxzlsk9xsszvjb1yszmzs0vlgawsssrq2vr5szmryaxilrmqh6kmgeschbqsmkk01okkpgep1wve5h61tsqjaudg5ve1pwhhbawi6c01davs3mnawrf5mefn+lewq8optmmdnbnx/ep+p3z+yfumhnwwroctxn7thjojfldhdjnkmq+drthigxdtill9kaorjjtz51y0ghif9q0rmkappvub7hvh+9qpylbodanuhio/pzogs9l4ns66ypqyekvff/y6rfqnbg9fksxogezpdskovyhtsvdtsyoutzrbihg+q+ydeaaubrirc7bnlx5mlqoc7zvsk+pcsxzcr0zti12ub7z6bgv0ruqotii6ben0ct6m56mz+pd+bhfz4zxzbb6bepzc/vyork=</latexit> it is sufficient (and necessary, in general) to include only the non-descendants of A in the definition of the predictor.

32 Examples A, X cannot be used. U can. A, Employed cannot be used. Prejudiced and Qualifications can. If it is judged that Prejudiced cannot be used, it should be labelled as a protected attributed.

33 How to Extract Unobserved Variables? Use the factual distribution to get a distribution over the unobserved variables by standard probabilistic conditioning. P(Unobserved Observed) Monte Carlo data augmentation approach: replace each data point in your training sample by a set of training points with the unobserved variables being filled by a Monte Carlo sample.

34 Workflow Regardless of the machine learning algorithm to be used, work with a domain expert to estimate a causal model of your data. It s a model of the world, not of your software. Choose any machine learning algorithm of interest, any black-box that takes as inputs observed and unobserved variables in your domain. Select a set of variables based on which sets respect counterfactual fairness. If necessary, infer unobserved variables from the observed ones.

35 Algorithm

36 Interpretation Extract causes of Y which are not mediators between A and Y. Find the best approximation to Y within the space of functions that exclude such mediators. Even if Y is unfair (A is a cause of it), by construction the predictor will be counterfactually fair.

37 Challenges Counterfactual fairness clarifies that algorithmic fairness in general is not explicitly modeling how the word becomes fairer with fair predictions. Even if our decision of giving a loan is fair, it doesn t mean that in aggregate the probability of a person of a particular demographic group won t have difficulties in repaying it (A still causes Y). The delayed impact of fair predictions is also a research topic see Liu et al.,

38 Workflow Regardless of the machine learning algorithm to be used, work with a domain expert to estimate a causal model of your data. It s a model of the world, not of your software. Choose any machine learning algorithm of interest, any black-box that takes as inputs observed and unobserved variables in your domain. Select a set of variables based on which sets respect counterfactual fairness. If necessary, infer unobserved variables from the observed ones.

39 Some Words of Caution Structural equations use unobserved variables. It is common that some of these variables are default choices based on some generic modeling assumption such as additive errors. Output = Signal + Noise Nature and society couldn t care less whether your mathematically convenient way of separating signal and noise is elegant or not.

40 Some Words of Caution That is, there are infinitely many structural equations V j = f j (V i, U j ) compatible with P(V i V j ) and P(V i do(v j )). Signal vs. noise must be determined by realworld assumptions ( simplicity assumptions, of the Ockham s razor type, can be sometimes adequate as long as caveats are advertised).

41 Some Words of Caution By now, there are several good papers on how to tackle fairness by generating unobserved variables which are independent of A, using assorted methods. U A Then off you go to plug-in U on a machine learning algorithm! X

42 However There are papers not causally motivated, which I find of difficult interpretation. Remember: there are infinitely many ways of extracting U. There are papers causally motivated, but which commit themselves to a domain-free family of structural equations. OK enough, but why would you do that? Counterfactual fairness emphasizes that the causal modeling step is separate from the prediction learning process. Finally, do beware of any paper that claims to do assumption-free extraction of causal latent factors. Those are selling you snake oil!

43 Interpretation of Counterfactuals But what does it mean to say had my race been different?? First, make sure to understand the difference between A and Perception of A : these can lead to conceptually different interpretations, even if the model stays the same. Without going in details, if those counterfactuals make you feel uneasy, just interpret them as comparing two different people who happen to match on the other things being equal factors. This is also related to fairness through awareness (Dwork et al., 2011,

44 Non-Counterfactual Causal Models Contrary to folk knowledge, causality does not require counterfactuals: the do operator is an way of comparing treatments without comparing individuals. However, if features X are affected by A, then in general there is no individual where P(Y = y X = x, do(a = a)) = P(Y = y X = x, do(a = a )) If features X are not affected by A, then we can show we do not need to explicitly model structural equations anyway!

45 The Upside Because structural causal models rely on unobserved variables, at least they can be partially falsified by eventually measuring some of those variables. Just keep in mind: while it is preposterous to say you have the causal model of a social process, you should (must?) be able to explain your assumptions to a regulator or a customer. Having passed testable implications, the remaining components of a counterfactual model should be understood as conjectures formulated according to the best of our knowledge. Such models should always be deemed provisional and prone to modifications.

46 Illustration The Law School Admission Council conducted a survey across 163 law schools in the United States It contains information on 21,790 law students such as their entrance exam scores (LSAT), their grade-point average (GPA) collected prior to law school, and their first year average grade (FYA). Task: predict if an applicant will have a high FYA Example of decision problem: make an offer

47 Setup I will present some simple causal models for this domain, which by no means I intend to sell as well-thought models. Their purpose is for illustration. We will fit real data to a model, then generate synthetic counterfactuals out of it. The point is to quantify to what extent a causally-oblivious method violates counterfactual fairness.

48 Two Models Fair K Fair Add

49 Predictive Error (Real Data) Comparison against Full (linear model with all variables) and Unaware (linear model without race and gender, but the other two predictors) Evaluation by root mean squared error

50 Fairness Violations (Simulated Counterfactuals)

51 Extension: Using Multiple Models We just saw two different counterfactual models that give different predictions despite being undistinguishable given the same data. This is OK assuming little difference between models, but we may have competing theories with some sizeable difference. We would like to be approximately counterfactually fair to all of them.

52 <latexit sha1_base64="6ftp5coo2gomwvzhxhd+kdhdw4c=">aaacm3icbvbdaxnbfj1ta61r29q++nixianysfsk+il040x6fmg0kwwidyc3yddz2xxmbiek/u996r/xqsg+kmvx/4ozaqra9sawh3puyeaejnfkcrhebuvlk49wh689qtx99nx9o7r54thlhzxulpnobcdbr1ozarnitz3ceqajpppk9ld0t87iopwzzzzjqzfiykihkshe6lepwvvzpeagl3vswdbc8jengcsavkjmuvm6mzhmytihd4bvovnedyhh3o/k0ia0y79ac5vhhhcfratsewu0+tvv8sctruqgpubnulgyc2+klpxudf6jc0c5ylmcuddtgym53ns+8zm89soahpn1xzdm1x8tu0ydm6sjn0yrx+6uv4oped2ch+97u2xygsnim4eghqboocwqbsqszd3xbkvv/q8gx2hrsq+54kui7q58nxzvnkowgx3are0dlopyey/fk1exkxgn9srh0rjticwf+c5+il/bzfajua5+34wubyvmlvgpwz+/koulag==</latexit> <latexit sha1_base64="6ftp5coo2gomwvzhxhd+kdhdw4c=">aaacm3icbvbdaxnbfj1ta61r29q++nixianysfsk+il040x6fmg0kwwidyc3yddz2xxmbiek/u996r/xqsg+kmvx/4ozaqra9sawh3puyeaejnfkcrhebuvlk49wh689qtx99nx9o7r54thlhzxulpnobcdbr1ozarnitz3ceqajpppk9ld0t87iopwzzzzjqzfiykihkshe6lepwvvzpeagl3vswdbc8jengcsavkjmuvm6mzhmytihd4bvovnedyhh3o/k0ia0y79ac5vhhhcfratsewu0+tvv8sctruqgpubnulgyc2+klpxudf6jc0c5ylmcuddtgym53ns+8zm89soahpn1xzdm1x8tu0ydm6sjn0yrx+6uv4oped2ch+97u2xygsnim4eghqboocwqbsqszd3xbkvv/q8gx2hrsq+54kui7q58nxzvnkowgx3are0dlopyey/fk1exkxgn9srh0rjticwf+c5+il/bzfajua5+34wubyvmlvgpwz+/koulag==</latexit> <latexit sha1_base64="6ftp5coo2gomwvzhxhd+kdhdw4c=">aaacm3icbvbdaxnbfj1ta61r29q++nixianysfsk+il040x6fmg0kwwidyc3yddz2xxmbiek/u996r/xqsg+kmvx/4ozaqra9sawh3puyeaejnfkcrhebuvlk49wh689qtx99nx9o7r54thlhzxulpnobcdbr1ozarnitz3ceqajpppk9ld0t87iopwzzzzjqzfiykihkshe6lepwvvzpeagl3vswdbc8jengcsavkjmuvm6mzhmytihd4bvovnedyhh3o/k0ia0y79ac5vhhhcfratsewu0+tvv8sctruqgpubnulgyc2+klpxudf6jc0c5ylmcuddtgym53ns+8zm89soahpn1xzdm1x8tu0ydm6sjn0yrx+6uv4oped2ch+97u2xygsnim4eghqboocwqbsqszd3xbkvv/q8gx2hrsq+54kui7q58nxzvnkowgx3are0dlopyey/fk1exkxgn9srh0rjticwf+c5+il/bzfajua5+34wubyvmlvgpwz+/koulag==</latexit> <latexit sha1_base64="6ftp5coo2gomwvzhxhd+kdhdw4c=">aaacm3icbvbdaxnbfj1ta61r29q++nixianysfsk+il040x6fmg0kwwidyc3yddz2xxmbiek/u996r/xqsg+kmvx/4ozaqra9sawh3puyeaejnfkcrhebuvlk49wh689qtx99nx9o7r54thlhzxulpnobcdbr1ozarnitz3ceqajpppk9ld0t87iopwzzzzjqzfiykihkshe6lepwvvzpeagl3vswdbc8jengcsavkjmuvm6mzhmytihd4bvovnedyhh3o/k0ia0y79ac5vhhhcfratsewu0+tvv8sctruqgpubnulgyc2+klpxudf6jc0c5ylmcuddtgym53ns+8zm89soahpn1xzdm1x8tu0ydm6sjn0yrx+6uv4oped2ch+97u2xygsnim4eghqboocwqbsqszd3xbkvv/q8gx2hrsq+54kui7q58nxzvnkowgx3are0dlopyey/fk1exkxgn9srh0rjticwf+c5+il/bzfajua5+34wubyvmlvgpwz+/koulag==</latexit> (ε, δ)- Counterfactual Fairness The following constraint provides a relaxation of counterfactual fairness: P ( Ŷ (a) Ŷ (a0 ) apple A = a, X = a) 1 The idea is to simultaneously satisfy such constraints according to different counterfactual models. It is not hard to show that this problem in general has no solution if ε = 0, hence the need for an approximate version.

53 Law School Revisited

54 COMPAS

55 A Different Direction: Interventions So far, we have solely discussed the creation of predictors. Ideally, we would like to destroy the pathways between A and Y, the outcome of interest. This is in general not possible. But let s assume we have an intervention with the ability of changing the contribution of A to Y. How is related to counterfactual fairness?

56 Imperfect Interventions and Interference We will assume two generalizations of the concept of intervention used so far. An intervention is represented generically as a set of (action) variables, which here I will denote as Z. We can define Z = 0 as the no action choice! Z 0 just means that one or more structural equations will change, not necessarily to a constant ( imperfect, or soft intervention).

57 Ideally Having available some Z = z which completely overrides the structural equation for Y to not depend on anything that starts on A. Z A Y

58 In Reality No such an intervention is typically available. And this is not a prediction problem anymore. What happens to Y? Setup: Assume Y is encoded so that high values are good. Model allows for interference: that is, treatment Z i given to person i might affect person j. How is this related to counterfactual fairness?

59 <latexit sha1_base64="xwnizrs8+qu/6klsoqwwnz6cx5y=">aaacznicbvfls8nagnzed3zfinjw8merfbqkiuhbwqecxwpwk00im+2mxbp5slsra9s/6c2zf3+gsdqdpg4sddpfspvn+glnulnwp6bpzm7nlywugcsrq2vr5kblscapilrbyh6lpo8l5syidcuup81eubz6nd77vdvcf36lqri4elt9hloh7kqsyasrtplm0dajser6pty1xjx2gd12bgmpgpddb4bgwlxh4bikq1nqn4+5caxgkbtfiv6tlkjduabh4dtwzkpvswranlenpiomqhvmh9oosrrssbgopwzzvqlcarakeu5hhpnkmmdswx3aymieqyrdqvhtcpyypq1bllitksju34kbdqxsh342mw8hy14u/ue1uhwcuwmwjamierlffkqcvax559bmghlf+xnbrldsruc6wgcisp/js7dlk0+tp5oabdxsh9pq1c2kjkw0g3brablrgbpc96iogoigl21jq2ib2re+pm/p2+nrxztkntef6pady1+w0g==</latexit> <latexit sha1_base64="xwnizrs8+qu/6klsoqwwnz6cx5y=">aaacznicbvfls8nagnzed3zfinjw8merfbqkiuhbwqecxwpwk00im+2mxbp5slsra9s/6c2zf3+gsdqdpg4sddpfspvn+glnulnwp6bpzm7nlywugcsrq2vr5kblscapilrbyh6lpo8l5syidcuup81eubz6nd77vdvcf36lqri4elt9hloh7kqsyasrtplm0dajser6pty1xjx2gd12bgmpgpddb4bgwlxh4bikq1nqn4+5caxgkbtfiv6tlkjduabh4dtwzkpvswranlenpiomqhvmh9oosrrssbgopwzzvqlcarakeu5hhpnkmmdswx3aymieqyrdqvhtcpyypq1bllitksju34kbdqxsh342mw8hy14u/ue1uhwcuwmwjamierlffkqcvax559bmghlf+xnbrldsruc6wgcisp/js7dlk0+tp5oabdxsh9pq1c2kjkw0g3brablrgbpc96iogoigl21jq2ib2re+pm/p2+nrxztkntef6pady1+w0g==</latexit> <latexit sha1_base64="xwnizrs8+qu/6klsoqwwnz6cx5y=">aaacznicbvfls8nagnzed3zfinjw8merfbqkiuhbwqecxwpwk00im+2mxbp5slsra9s/6c2zf3+gsdqdpg4sddpfspvn+glnulnwp6bpzm7nlywugcsrq2vr5kblscapilrbyh6lpo8l5syidcuup81eubz6nd77vdvcf36lqri4elt9hloh7kqsyasrtplm0dajser6pty1xjx2gd12bgmpgpddb4bgwlxh4bikq1nqn4+5caxgkbtfiv6tlkjduabh4dtwzkpvswranlenpiomqhvmh9oosrrssbgopwzzvqlcarakeu5hhpnkmmdswx3aymieqyrdqvhtcpyypq1bllitksju34kbdqxsh342mw8hy14u/ue1uhwcuwmwjamierlffkqcvax559bmghlf+xnbrldsruc6wgcisp/js7dlk0+tp5oabdxsh9pq1c2kjkw0g3brablrgbpc96iogoigl21jq2ib2re+pm/p2+nrxztkntef6pady1+w0g==</latexit> <latexit sha1_base64="xwnizrs8+qu/6klsoqwwnz6cx5y=">aaacznicbvfls8nagnzed3zfinjw8merfbqkiuhbwqecxwpwk00im+2mxbp5slsra9s/6c2zf3+gsdqdpg4sddpfspvn+glnulnwp6bpzm7nlywugcsrq2vr5kblscapilrbyh6lpo8l5syidcuup81eubz6nd77vdvcf36lqri4elt9hloh7kqsyasrtplm0dajser6pty1xjx2gd12bgmpgpddb4bgwlxh4bikq1nqn4+5caxgkbtfiv6tlkjduabh4dtwzkpvswranlenpiomqhvmh9oosrrssbgopwzzvqlcarakeu5hhpnkmmdswx3aymieqyrdqvhtcpyypq1bllitksju34kbdqxsh342mw8hy14u/ue1uhwcuwmwjamierlffkqcvax559bmghlf+xnbrldsruc6wgcisp/js7dlk0+tp5oabdxsh9pq1c2kjkw0g3brablrgbpc96iogoigl21jq2ib2re+pm/p2+nrxztkntef6pady1+w0g==</latexit> <latexit sha1_base64="4c9mlclgudbw9ia5ysx9ikequku=">aaace3icbvfdaxnbfj1dv2r8ivooymugidkgxvhsg0krid5wmg0kuwx3j3fbobozy8ysnk77j/xpffof+ci42erb0x4yojxzd3fuvvmlphvr9csir1y9dv3g1s3erdt37t7r339wamvacjqiupvmmqeljtvnnhskppuhldjfr9np+6v/9i2mlax+4hyvpquea5llgc5lvp8zqfwctgzqunnlcnqnwqyfzl+5hcgx27cscvj+liefkma+l/awomva0tmuu9ibzexwi3pjsou2vpnig4ndtou3kdisex8qjamocjheazjgaxzw/nkyl0vdkhzcobwzokpc2qbxuihqe0ltqujxisc081rjqtztut218nqrc8hl45920kn/jhosrf0uma9cjmm3vav4mterxb6bnljxtsmtvo3ywoeryxkimetdwqmfjyim9h8fcyl+ds6fq+exeg+ofjecvhjh0tj+/hkw9269ji32id1hixaz12ypfwihbmie+x08dobbkpgtdsln4faqnazwmyfsp4sv/giwtbob</latexit> <latexit sha1_base64="4c9mlclgudbw9ia5ysx9ikequku=">aaace3icbvfdaxnbfj1dv2r8ivooymugidkgxvhsg0krid5wmg0kuwx3j3fbobozy8ysnk77j/xpffof+ci42erb0x4yojxzd3fuvvmlphvr9csir1y9dv3g1s3erdt37t7r339wamvacjqiupvmmqeljtvnnhskppuhldjfr9np+6v/9i2mlax+4hyvpquea5llgc5lvp8zqfwctgzqunnlcnqnwqyfzl+5hcgx27cscvj+liefkma+l/awomva0tmuu9ibzexwi3pjsou2vpnig4ndtou3kdisex8qjamocjheazjgaxzw/nkyl0vdkhzcobwzokpc2qbxuihqe0ltqujxisc081rjqtztut218nqrc8hl45920kn/jhosrf0uma9cjmm3vav4mterxb6bnljxtsmtvo3ywoeryxkimetdwqmfjyim9h8fcyl+ds6fq+exeg+ofjecvhjh0tj+/hkw9269ji32id1hixaz12ypfwihbmie+x08dobbkpgtdsln4faqnazwmyfsp4sv/giwtbob</latexit> <latexit sha1_base64="4c9mlclgudbw9ia5ysx9ikequku=">aaace3icbvfdaxnbfj1dv2r8ivooymugidkgxvhsg0krid5wmg0kuwx3j3fbobozy8ysnk77j/xpffof+ci42erb0x4yojxzd3fuvvmlphvr9csir1y9dv3g1s3erdt37t7r339wamvacjqiupvmmqeljtvnnhskppuhldjfr9np+6v/9i2mlax+4hyvpquea5llgc5lvp8zqfwctgzqunnlcnqnwqyfzl+5hcgx27cscvj+liefkma+l/awomva0tmuu9ibzexwi3pjsou2vpnig4ndtou3kdisex8qjamocjheazjgaxzw/nkyl0vdkhzcobwzokpc2qbxuihqe0ltqujxisc081rjqtztut218nqrc8hl45920kn/jhosrf0uma9cjmm3vav4mterxb6bnljxtsmtvo3ywoeryxkimetdwqmfjyim9h8fcyl+ds6fq+exeg+ofjecvhjh0tj+/hkw9269ji32id1hixaz12ypfwihbmie+x08dobbkpgtdsln4faqnazwmyfsp4sv/giwtbob</latexit> <latexit sha1_base64="4c9mlclgudbw9ia5ysx9ikequku=">aaace3icbvfdaxnbfj1dv2r8ivooymugidkgxvhsg0krid5wmg0kuwx3j3fbobozy8ysnk77j/xpffof+ci42erb0x4yojxzd3fuvvmlphvr9csir1y9dv3g1s3erdt37t7r339wamvacjqiupvmmqeljtvnnhskppuhldjfr9np+6v/9i2mlax+4hyvpquea5llgc5lvp8zqfwctgzqunnlcnqnwqyfzl+5hcgx27cscvj+liefkma+l/awomva0tmuu9ibzexwi3pjsou2vpnig4ndtou3kdisex8qjamocjheazjgaxzw/nkyl0vdkhzcobwzokpc2qbxuihqe0ltqujxisc081rjqtztut218nqrc8hl45920kn/jhosrf0uma9cjmm3vav4mterxb6bnljxtsmtvo3ywoeryxkimetdwqmfjyim9h8fcyl+ds6fq+exeg+ofjecvhjh0tj+/hkw9269ji32id1hixaz12ypfwihbmie+x08dobbkpgtdsln4faqnazwmyfsp4sv/giwtbob</latexit> <latexit sha1_base64="we8pnaxqrv0xhrakialrilrid8q=">aaac23icbvjdb9mwfhxcx0b5wifhxq6oyeokqgqhdvfn2oyqpa6jbkv1sg5cz7pmofnsoluhvpaaqrzyx3jjb/alcnpkqmevlb2fc+71vbatqkljw/cn51+6foxq2vq1zvubn29tdg/fotr5vxix5lnky1gcriipxdbkq8sokavmirjhyenzvj96l0ojc/3gtgsrz3iszso5wkex7i+a4tmrzywkgkpjbk0am6ybokbkwc13ouadfkgh4hz2jengxfgtk1ulxqqzrxq+unhjenyamqxgnifntmauhizc1elswv7yhllamx3bd+cmvovpxbjv4gz2gepcv1kisnd+vnvpjljzlhrqswrzb21hav6iuocbqkvenmhrwv4acs5ewxiihfwp2+mwbi/sh/oaiybagh5zxghr/qctnfez0jyrngychywnayyt5eo0hvozusa/xwmxdlbjjkxcz9+mgqeomybr1s1tyc7+nvfjzsw0s5yzhccsai35p21c2frpxetdvfzovjgorrtyhnqhhoksbbdq6gdyurpegz9gidy679beqrq68kvw+lgfhf3o9zpe7v7yotbjpxkfbjgibjnd8oockchh3sj76h32vvix/8n/6n9bwh1vmxox/bp+998ayt1q</latexit> <latexit sha1_base64="we8pnaxqrv0xhrakialrilrid8q=">aaac23icbvjdb9mwfhxcx0b5wifhxq6oyeokqgqhdvfn2oyqpa6jbkv1sg5cz7pmofnsoluhvpaaqrzyx3jjb/alcnpkqmevlb2fc+71vbatqkljw/cn51+6foxq2vq1zvubn29tdg/fotr5vxix5lnky1gcriipxdbkq8sokavmirjhyenzvj96l0ojc/3gtgsrz3iszso5wkex7i+a4tmrzywkgkpjbk0am6ybokbkwc13ouadfkgh4hz2jengxfgtk1ulxqqzrxq+unhjenyamqxgnifntmauhizc1elswv7yhllamx3bd+cmvovpxbjv4gz2gepcv1kisnd+vnvpjljzlhrqswrzb21hav6iuocbqkvenmhrwv4acs5ewxiihfwp2+mwbi/sh/oaiybagh5zxghr/qctnfez0jyrngychywnayyt5eo0hvozusa/xwmxdlbjjkxcz9+mgqeomybr1s1tyc7+nvfjzsw0s5yzhccsai35p21c2frpxetdvfzovjgorrtyhnqhhoksbbdq6gdyurpegz9gidy679beqrq68kvw+lgfhf3o9zpe7v7yotbjpxkfbjgibjnd8oockchh3sj76h32vvix/8n/6n9bwh1vmxox/bp+998ayt1q</latexit> <latexit sha1_base64="we8pnaxqrv0xhrakialrilrid8q=">aaac23icbvjdb9mwfhxcx0b5wifhxq6oyeokqgqhdvfn2oyqpa6jbkv1sg5cz7pmofnsoluhvpaaqrzyx3jjb/alcnpkqmevlb2fc+71vbatqkljw/cn51+6foxq2vq1zvubn29tdg/fotr5vxix5lnky1gcriipxdbkq8sokavmirjhyenzvj96l0ojc/3gtgsrz3iszso5wkex7i+a4tmrzywkgkpjbk0am6ybokbkwc13ouadfkgh4hz2jengxfgtk1ulxqqzrxq+unhjenyamqxgnifntmauhizc1elswv7yhllamx3bd+cmvovpxbjv4gz2gepcv1kisnd+vnvpjljzlhrqswrzb21hav6iuocbqkvenmhrwv4acs5ewxiihfwp2+mwbi/sh/oaiybagh5zxghr/qctnfez0jyrngychywnayyt5eo0hvozusa/xwmxdlbjjkxcz9+mgqeomybr1s1tyc7+nvfjzsw0s5yzhccsai35p21c2frpxetdvfzovjgorrtyhnqhhoksbbdq6gdyurpegz9gidy679beqrq68kvw+lgfhf3o9zpe7v7yotbjpxkfbjgibjnd8oockchh3sj76h32vvix/8n/6n9bwh1vmxox/bp+998ayt1q</latexit> <latexit sha1_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixv1l86vq1rn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlt1eldvesarh6kc0vo5wtdmajnchizrxwyasblykjn1yqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha1_base64="z/0gulhd2yytxjooab1lgkjlzsa=">aaac0hicbvllbtqwfhxcq4qca1s2v1tqikwjpbsqo0ouhgbzjkydna7wjcdprtpogjuomyfswiaqx8eo3+alccyjavouzon4npu201jjy6pop+dfunzl6rwn68gnzzu3bg/ubb6aoq64gpncfdukrsou1gjspvviulyc81sjo/t0racffrcvkyv+a+elshi81jkthk2j2oaxzfgcnqswh0dvrlamhaxtlvbt56yre3h7xj+ki3b+9irn4ex0hzm7/s2dxsmkhynsmwl7geygmfnccyytlw9torpx9rqobuspdczqdl2d0xryhquispzbc6c60hweigqcf6e9co643fy6tvh38aouo6youcnababs9w1yvlafglwycbiasid/0gybg2xfw2hpcbhek7bfvnbabj/orob1lrtlco2zxlfpkwyrk7ksbubri0rkp3gspg5qzivjmuxbtpdamtnwrbqjlszzvymazi2z56nz7ayw61ph/k+b1jz7kjrsl7uvmvefslqblab7ajjjsncr5g4gr6trffgjvsit+w7deul1ks+cw91hha3jnxhzipfifbjdyvkypcovyqeze+5nve/ef++rn/if/w/9unxvtbe75b/zv/8gccnccq==</latexit> <latexit sha1_base64="z/0gulhd2yytxjooab1lgkjlzsa=">aaac0hicbvllbtqwfhxcq4qca1s2v1tqikwjpbsqo0ouhgbzjkydna7wjcdprtpogjuomyfswiaqx8eo3+alccyjavouzon4npu201jjy6pop+dfunzl6rwn68gnzzu3bg/ubb6aoq64gpncfdukrsou1gjspvviulyc81sjo/t0racffrcvkyv+a+elshi81jkthk2j2oaxzfgcnqswh0dvrlamhaxtlvbt56yre3h7xj+ki3b+9irn4ex0hzm7/s2dxsmkhynsmwl7geygmfnccyytlw9torpx9rqobuspdczqdl2d0xryhquispzbc6c60hweigqcf6e9co643fy6tvh38aouo6youcnababs9w1yvlafglwycbiasid/0gybg2xfw2hpcbhek7bfvnbabj/orob1lrtlco2zxlfpkwyrk7ksbubri0rkp3gspg5qzivjmuxbtpdamtnwrbqjlszzvymazi2z56nz7ayw61ph/k+b1jz7kjrsl7uvmvefslqblab7ajjjsncr5g4gr6trffgjvsit+w7deul1ks+cw91hha3jnxhzipfifbjdyvkypcovyqeze+5nve/ef++rn/if/w/9unxvtbe75b/zv/8gccnccq==</latexit> <latexit sha1_base64="0tqt7/ceh+4oxoofio+lz2mzclc=">aaac23icbvjnb9qwehxcvwlfcxy5jfhbixstei4gvpxaigthirhtonwwjl5va9vx0thb3q3hwggeuplhupe3+au4m5walsnzen7vzxjgdlooawwu/ft8cxcvxb6yctw4dv3gzvu923cotf6vxix4rvjynkirsmoxstiqms5kgvmqxgf68rzvd9+l0shcv7hzqiqzhmk5kxyto1jvf83wjnulfoda1ts3joqf0w1qu2wslttx804/penwpnucpvbi8pbjrw43g8ujch8o7dij24bmhjbewjmme1cpu6fqcfosl2xdkq3mwa7cocp14guxrju4ht2gotdvloosnl/vxjojbjadti1wbf6w7wiwl6gu4czqqbsgosrydueumthctatdp2wdgpx60sbabpwh8qr0ysr2we8hnea8yos2xkexkzgqbfjjasvxoglozusb/aspxmrbjzkwsb18mwyeogykrlw3tiul+3dgjzkx8yx1znyas6615p+0swvnt5na6qkyqvpuofmlwobqpjrmzsm4vxmhkjfs9qr8gevk1n2h9hli9zhpg4phgzgaxk+j/s7e6jo2yd1yn2yrmdwho+qv2scjwr2x99h77h3xe/+t/9x/1ll9b5vzl/wt/vff/3vdta==</latexit> <latexit sha1_base64="we8pnaxqrv0xhrakialrilrid8q=">aaac23icbvjdb9mwfhxcx0b5wifhxq6oyeokqgqhdvfn2oyqpa6jbkv1sg5cz7pmofnsoluhvpaaqrzyx3jjb/alcnpkqmevlb2fc+71vbatqkljw/cn51+6foxq2vq1zvubn29tdg/fotr5vxix5lnky1gcriipxdbkq8sokavmirjhyenzvj96l0ojc/3gtgsrz3iszso5wkex7i+a4tmrzywkgkpjbk0am6ybokbkwc13ouadfkgh4hz2jengxfgtk1ulxqqzrxq+unhjenyamqxgnifntmauhizc1elswv7yhllamx3bd+cmvovpxbjv4gz2gepcv1kisnd+vnvpjljzlhrqswrzb21hav6iuocbqkvenmhrwv4acs5ewxiihfwp2+mwbi/sh/oaiybagh5zxghr/qctnfez0jyrngychywnayyt5eo0hvozusa/xwmxdlbjjkxcz9+mgqeomybr1s1tyc7+nvfjzsw0s5yzhccsai35p21c2frpxetdvfzovjgorrtyhnqhhoksbbdq6gdyurpegz9gidy679beqrq68kvw+lgfhf3o9zpe7v7yotbjpxkfbjgibjnd8oockchh3sj76h32vvix/8n/6n9bwh1vmxox/bp+998ayt1q</latexit> <latexit sha1_base64="we8pnaxqrv0xhrakialrilrid8q=">aaac23icbvjdb9mwfhxcx0b5wifhxq6oyeokqgqhdvfn2oyqpa6jbkv1sg5cz7pmofnsoluhvpaaqrzyx3jjb/alcnpkqmevlb2fc+71vbatqkljw/cn51+6foxq2vq1zvubn29tdg/fotr5vxix5lnky1gcriipxdbkq8sokavmirjhyenzvj96l0ojc/3gtgsrz3iszso5wkex7i+a4tmrzywkgkpjbk0am6ybokbkwc13ouadfkgh4hz2jengxfgtk1ulxqqzrxq+unhjenyamqxgnifntmauhizc1elswv7yhllamx3bd+cmvovpxbjv4gz2gepcv1kisnd+vnvpjljzlhrqswrzb21hav6iuocbqkvenmhrwv4acs5ewxiihfwp2+mwbi/sh/oaiybagh5zxghr/qctnfez0jyrngychywnayyt5eo0hvozusa/xwmxdlbjjkxcz9+mgqeomybr1s1tyc7+nvfjzsw0s5yzhccsai35p21c2frpxetdvfzovjgorrtyhnqhhoksbbdq6gdyurpegz9gidy679beqrq68kvw+lgfhf3o9zpe7v7yotbjpxkfbjgibjnd8oockchh3sj76h32vvix/8n/6n9bwh1vmxox/bp+998ayt1q</latexit> <latexit sha1_base64="we8pnaxqrv0xhrakialrilrid8q=">aaac23icbvjdb9mwfhxcx0b5wifhxq6oyeokqgqhdvfn2oyqpa6jbkv1sg5cz7pmofnsoluhvpaaqrzyx3jjb/alcnpkqmevlb2fc+71vbatqkljw/cn51+6foxq2vq1zvubn29tdg/fotr5vxix5lnky1gcriipxdbkq8sokavmirjhyenzvj96l0ojc/3gtgsrz3iszso5wkex7i+a4tmrzywkgkpjbk0am6ybokbkwc13ouadfkgh4hz2jengxfgtk1ulxqqzrxq+unhjenyamqxgnifntmauhizc1elswv7yhllamx3bd+cmvovpxbjv4gz2gepcv1kisnd+vnvpjljzlhrqswrzb21hav6iuocbqkvenmhrwv4acs5ewxiihfwp2+mwbi/sh/oaiybagh5zxghr/qctnfez0jyrngychywnayyt5eo0hvozusa/xwmxdlbjjkxcz9+mgqeomybr1s1tyc7+nvfjzsw0s5yzhccsai35p21c2frpxetdvfzovjgorrtyhnqhhoksbbdq6gdyurpegz9gidy679beqrq68kvw+lgfhf3o9zpe7v7yotbjpxkfbjgibjnd8oockchh3sj76h32vvix/8n/6n9bwh1vmxox/bp+998ayt1q</latexit> <latexit sha1_base64="we8pnaxqrv0xhrakialrilrid8q=">aaac23icbvjdb9mwfhxcx0b5wifhxq6oyeokqgqhdvfn2oyqpa6jbkv1sg5cz7pmofnsoluhvpaaqrzyx3jjb/alcnpkqmevlb2fc+71vbatqkljw/cn51+6foxq2vq1zvubn29tdg/fotr5vxix5lnky1gcriipxdbkq8sokavmirjhyenzvj96l0ojc/3gtgsrz3iszso5wkex7i+a4tmrzywkgkpjbk0am6ybokbkwc13ouadfkgh4hz2jengxfgtk1ulxqqzrxq+unhjenyamqxgnifntmauhizc1elswv7yhllamx3bd+cmvovpxbjv4gz2gepcv1kisnd+vnvpjljzlhrqswrzb21hav6iuocbqkvenmhrwv4acs5ewxiihfwp2+mwbi/sh/oaiybagh5zxghr/qctnfez0jyrngychywnayyt5eo0hvozusa/xwmxdlbjjkxcz9+mgqeomybr1s1tyc7+nvfjzsw0s5yzhccsai35p21c2frpxetdvfzovjgorrtyhnqhhoksbbdq6gdyurpegz9gidy679beqrq68kvw+lgfhf3o9zpe7v7yotbjpxkfbjgibjnd8oockchh3sj76h32vvix/8n/6n9bwh1vmxox/bp+998ayt1q</latexit> <latexit sha1_base64="we8pnaxqrv0xhrakialrilrid8q=">aaac23icbvjdb9mwfhxcx0b5wifhxq6oyeokqgqhdvfn2oyqpa6jbkv1sg5cz7pmofnsoluhvpaaqrzyx3jjb/alcnpkqmevlb2fc+71vbatqkljw/cn51+6foxq2vq1zvubn29tdg/fotr5vxix5lnky1gcriipxdbkq8sokavmirjhyenzvj96l0ojc/3gtgsrz3iszso5wkex7i+a4tmrzywkgkpjbk0am6ybokbkwc13ouadfkgh4hz2jengxfgtk1ulxqqzrxq+unhjenyamqxgnifntmauhizc1elswv7yhllamx3bd+cmvovpxbjv4gz2gepcv1kisnd+vnvpjljzlhrqswrzb21hav6iuocbqkvenmhrwv4acs5ewxiihfwp2+mwbi/sh/oaiybagh5zxghr/qctnfez0jyrngychywnayyt5eo0hvozusa/xwmxdlbjjkxcz9+mgqeomybr1s1tyc7+nvfjzsw0s5yzhccsai35p21c2frpxetdvfzovjgorrtyhnqhhoksbbdq6gdyurpegz9gidy679beqrq68kvw+lgfhf3o9zpe7v7yotbjpxkfbjgibjnd8oockchh3sj76h32vvix/8n/6n9bwh1vmxox/bp+998ayt1q</latexit> <latexit sha1_base64="we8pnaxqrv0xhrakialrilrid8q=">aaac23icbvjdb9mwfhxcx0b5wifhxq6oyeokqgqhdvfn2oyqpa6jbkv1sg5cz7pmofnsoluhvpaaqrzyx3jjb/alcnpkqmevlb2fc+71vbatqkljw/cn51+6foxq2vq1zvubn29tdg/fotr5vxix5lnky1gcriipxdbkq8sokavmirjhyenzvj96l0ojc/3gtgsrz3iszso5wkex7i+a4tmrzywkgkpjbk0am6ybokbkwc13ouadfkgh4hz2jengxfgtk1ulxqqzrxq+unhjenyamqxgnifntmauhizc1elswv7yhllamx3bd+cmvovpxbjv4gz2gepcv1kisnd+vnvpjljzlhrqswrzb21hav6iuocbqkvenmhrwv4acs5ewxiihfwp2+mwbi/sh/oaiybagh5zxghr/qctnfez0jyrngychywnayyt5eo0hvozusa/xwmxdlbjjkxcz9+mgqeomybr1s1tyc7+nvfjzsw0s5yzhccsai35p21c2frpxetdvfzovjgorrtyhnqhhoksbbdq6gdyurpegz9gidy679beqrq68kvw+lgfhf3o9zpe7v7yotbjpxkfbjgibjnd8oockchh3sj76h32vvix/8n/6n9bwh1vmxox/bp+998ayt1q</latexit> Optimization Problem and Constraints Main family of constraints: E[Y i (a i, z) A i = a i,x i = x i ] E[Y i (a 0, z) A i = a i,x i = x i ] {z } G ia 0 < (as opposed to E[Y i (a i, z) A i = a i,x i = x i ] E[Y i (a 0, z) A i = a i,x i = x i ] < ) max z 1,...,z n s.t., nx i=1 nx i=1 E[Y i (z) A i = a i,x i = x i ] z i apple B G ia 0 apple 8a 0 2 A,i2 1,...,n,

60 Intuitive Toy Example Protected attribute A is such that A in {b, w}, X is some quantitative measure of professional competence, and Y is a measure of wealth in 5 years time. Z i = 1 means individual i gets a subsidy to move to a neighborhood with better transport links.

61 <latexit sha1_base64="g6o0fzji/vpizil8s528m+cdgfg=">aaacjxicbvbns8naen3ur1q/oh69lbahiprefd0ovl3oryjpq00im+22xbr5yhejlna/48w/4swdrqrp/hu3aq7aora7jzdvmjnnrywkarhfwmfufmfxqbhcwlldw9/qn7caiow5jhyowchbhhke0ybykkpgwhenypcyaxqdq7tefcrc0dc4k8oiod7qbbrlmzkkcvwze5fcc9hs/we0demha8dztix1iyc2j2tf85kbueuiez/tk4nl0pkrl42qkqwcbwyoyicpuqup7u6iy58eejmkrns0iukkieukgrmv7fiqcoeb6pg2ggfs850ku3ie9xttgd2qqxdimlg/oxlkczh0pavmnxbttzt8r9aozffuswgqxzieedkogzmoq5habjuueyzzuageovw7qtxhhggpje1nmkdpngwnw6ppvm3bo3ltmrejchbalqgae5yagrggdwabdj7bk3ghy+1fe9m+tm+jtkdlpdvgt2jfp7p3oho=</latexit> <latexit sha1_base64="g6o0fzji/vpizil8s528m+cdgfg=">aaacjxicbvbns8naen3ur1q/oh69lbahiprefd0ovl3oryjpq00im+22xbr5yhejlna/48w/4swdrqrp/hu3aq7aora7jzdvmjnnrywkarhfwmfufmfxqbhcwlldw9/qn7caiow5jhyowchbhhke0ybykkpgwhenypcyaxqdq7tefcrc0dc4k8oiod7qbbrlmzkkcvwze5fcc9hs/we0demha8dztix1iyc2j2tf85kbueuiez/tk4nl0pkrl42qkqwcbwyoyicpuqup7u6iy58eejmkrns0iukkieukgrmv7fiqcoeb6pg2ggfs850ku3ie9xttgd2qqxdimlg/oxlkczh0pavmnxbttzt8r9aozffuswgqxzieedkogzmoq5habjuueyzzuageovw7qtxhhggpje1nmkdpngwnw6ppvm3bo3ltmrejchbalqgae5yagrggdwabdj7bk3ghy+1fe9m+tm+jtkdlpdvgt2jfp7p3oho=</latexit> <latexit sha1_base64="g6o0fzji/vpizil8s528m+cdgfg=">aaacjxicbvbns8naen3ur1q/oh69lbahiprefd0ovl3oryjpq00im+22xbr5yhejlna/48w/4swdrqrp/hu3aq7aora7jzdvmjnnrywkarhfwmfufmfxqbhcwlldw9/qn7caiow5jhyowchbhhke0ybykkpgwhenypcyaxqdq7tefcrc0dc4k8oiod7qbbrlmzkkcvwze5fcc9hs/we0demha8dztix1iyc2j2tf85kbueuiez/tk4nl0pkrl42qkqwcbwyoyicpuqup7u6iy58eejmkrns0iukkieukgrmv7fiqcoeb6pg2ggfs850ku3ie9xttgd2qqxdimlg/oxlkczh0pavmnxbttzt8r9aozffuswgqxzieedkogzmoq5habjuueyzzuageovw7qtxhhggpje1nmkdpngwnw6ppvm3bo3ltmrejchbalqgae5yagrggdwabdj7bk3ghy+1fe9m+tm+jtkdlpdvgt2jfp7p3oho=</latexit> <latexit sha1_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixv1l86vq1rn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlt1eldvesarh6kc0vo5wtdmajnchizrxwyasblykjn1yqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha1_base64="5cglw4oc6hfwrkutbeluh14wmoc=">aaacgnicbvdlsgmxfl1tx7vwhd26crahipsmilpqunzoroj9afugtjq2ozkhsuypq3/gjb/ixovfbp/gzlqlbt2q5hducffe40wck43xt5vbwl5zxcuvfzakm1vb9k6xrsjyulajoqhl0yokcr6wmuzasgykgfe9wrre8dqtn56yvdwm7vuoyh2f9ape45roi7n2+ypl0qvqmvsiorg/zuqke9ua+0yhtk/0wpos23h5kjm/hxplzeuf1y7hcs6afokziywyoerak3y3plhpak0fuarl4eh3eii1p4knc+1ysyjqiemzlqebmf07sbblgb0ypyt6otqn0chtf/9iik/uypemm51yzdds8b9ak9a9s07cgyjwlkdtrr1yib2indlu5zjrluagecq5mrxrazgeahnsgoizv/iiqr9xhfxx7jdkyq/2oqwonmil3eavakdhbd7gaybwq/vufu7jylmz3hbhd6yvh9ponuo=</latexit> <latexit sha1_base64="5cglw4oc6hfwrkutbeluh14wmoc=">aaacgnicbvdlsgmxfl1tx7vwhd26crahipsmilpqunzoroj9afugtjq2ozkhsuypq3/gjb/ixovfbp/gzlqlbt2q5hducffe40wck43xt5vbwl5zxcuvfzakm1vb9k6xrsjyulajoqhl0yokcr6wmuzasgykgfe9wrre8dqtn56yvdwm7vuoyh2f9ape45roi7n2+ypl0qvqmvsiorg/zuqke9ua+0yhtk/0wpos23h5kjm/hxplzeuf1y7hcs6afokziywyoerak3y3plhpak0fuarl4eh3eii1p4knc+1ysyjqiemzlqebmf07sbblgb0ypyt6otqn0chtf/9iik/uypemm51yzdds8b9ak9a9s07cgyjwlkdtrr1yib2indlu5zjrluagecq5mrxrazgeahnsgoizv/iiqr9xhfxx7jdkyq/2oqwonmil3eavakdhbd7gaybwq/vufu7jylmz3hbhd6yvh9ponuo=</latexit> <latexit sha1_base64="6vdkphhjm2k4gyz6mkhn0+38qme=">aaacjxicbvdlsgmxfm34rpvvdekmwiskudkc6ekh6kz3fexdo8oqstntaozbklhk0j9x46+4cwerwzw/ymy6c229knzduedy7z1uxjluch0zc/mli0vlhzxi6tr6xmzpa7spw1gq2iahd0xbxzjyftcgyortdiqo9l1ow+7gkq23hqmqlazu1dcito97afmywupttuns3mhwhlb1fwhnhb4ycjxlszgfsmj5wpvdn7kzvs4y8dobljqcvnrkzvrfwcbzyoagdpkoo6wx1q1j7nnaey6l7jgounachwke01hriiwnmbnghu1ogga9306yk0dwxznd6ivcv0dbjp3dkwbfyqhvamw6szyuper/tu6svfm7yueukxqqysav5lcfmlumdpmgrpghbpgipnefpi8fjkobm5pgtp88c5phvrnvzvturl3mdhtaltgdfwcce1ad16aogocaz/ak3shyedheja/jcykdm/kehfanjo8fsregdg==</latexit> <latexit sha1_base64="g6o0fzji/vpizil8s528m+cdgfg=">aaacjxicbvbns8naen3ur1q/oh69lbahiprefd0ovl3oryjpq00im+22xbr5yhejlna/48w/4swdrqrp/hu3aq7aora7jzdvmjnnrywkarhfwmfufmfxqbhcwlldw9/qn7caiow5jhyowchbhhke0ybykkpgwhenypcyaxqdq7tefcrc0dc4k8oiod7qbbrlmzkkcvwze5fcc9hs/we0demha8dztix1iyc2j2tf85kbueuiez/tk4nl0pkrl42qkqwcbwyoyicpuqup7u6iy58eejmkrns0iukkieukgrmv7fiqcoeb6pg2ggfs850ku3ie9xttgd2qqxdimlg/oxlkczh0pavmnxbttzt8r9aozffuswgqxzieedkogzmoq5habjuueyzzuageovw7qtxhhggpje1nmkdpngwnw6ppvm3bo3ltmrejchbalqgae5yagrggdwabdj7bk3ghy+1fe9m+tm+jtkdlpdvgt2jfp7p3oho=</latexit> <latexit sha1_base64="g6o0fzji/vpizil8s528m+cdgfg=">aaacjxicbvbns8naen3ur1q/oh69lbahiprefd0ovl3oryjpq00im+22xbr5yhejlna/48w/4swdrqrp/hu3aq7aora7jzdvmjnnrywkarhfwmfufmfxqbhcwlldw9/qn7caiow5jhyowchbhhke0ybykkpgwhenypcyaxqdq7tefcrc0dc4k8oiod7qbbrlmzkkcvwze5fcc9hs/we0demha8dztix1iyc2j2tf85kbueuiez/tk4nl0pkrl42qkqwcbwyoyicpuqup7u6iy58eejmkrns0iukkieukgrmv7fiqcoeb6pg2ggfs850ku3ie9xttgd2qqxdimlg/oxlkczh0pavmnxbttzt8r9aozffuswgqxzieedkogzmoq5habjuueyzzuageovw7qtxhhggpje1nmkdpngwnw6ppvm3bo3ltmrejchbalqgae5yagrggdwabdj7bk3ghy+1fe9m+tm+jtkdlpdvgt2jfp7p3oho=</latexit> <latexit sha1_base64="g6o0fzji/vpizil8s528m+cdgfg=">aaacjxicbvbns8naen3ur1q/oh69lbahiprefd0ovl3oryjpq00im+22xbr5yhejlna/48w/4swdrqrp/hu3aq7aora7jzdvmjnnrywkarhfwmfufmfxqbhcwlldw9/qn7caiow5jhyowchbhhke0ybykkpgwhenypcyaxqdq7tefcrc0dc4k8oiod7qbbrlmzkkcvwze5fcc9hs/we0demha8dztix1iyc2j2tf85kbueuiez/tk4nl0pkrl42qkqwcbwyoyicpuqup7u6iy58eejmkrns0iukkieukgrmv7fiqcoeb6pg2ggfs850ku3ie9xttgd2qqxdimlg/oxlkczh0pavmnxbttzt8r9aozffuswgqxzieedkogzmoq5habjuueyzzuageovw7qtxhhggpje1nmkdpngwnw6ppvm3bo3ltmrejchbalqgae5yagrggdwabdj7bk3ghy+1fe9m+tm+jtkdlpdvgt2jfp7p3oho=</latexit> <latexit sha1_base64="g6o0fzji/vpizil8s528m+cdgfg=">aaacjxicbvbns8naen3ur1q/oh69lbahiprefd0ovl3oryjpq00im+22xbr5yhejlna/48w/4swdrqrp/hu3aq7aora7jzdvmjnnrywkarhfwmfufmfxqbhcwlldw9/qn7caiow5jhyowchbhhke0ybykkpgwhenypcyaxqdq7tefcrc0dc4k8oiod7qbbrlmzkkcvwze5fcc9hs/we0demha8dztix1iyc2j2tf85kbueuiez/tk4nl0pkrl42qkqwcbwyoyicpuqup7u6iy58eejmkrns0iukkieukgrmv7fiqcoeb6pg2ggfs850ku3ie9xttgd2qqxdimlg/oxlkczh0pavmnxbttzt8r9aozffuswgqxzieedkogzmoq5habjuueyzzuageovw7qtxhhggpje1nmkdpngwnw6ppvm3bo3ltmrejchbalqgae5yagrggdwabdj7bk3ghy+1fe9m+tm+jtkdlpdvgt2jfp7p3oho=</latexit> <latexit sha1_base64="g6o0fzji/vpizil8s528m+cdgfg=">aaacjxicbvbns8naen3ur1q/oh69lbahiprefd0ovl3oryjpq00im+22xbr5yhejlna/48w/4swdrqrp/hu3aq7aora7jzdvmjnnrywkarhfwmfufmfxqbhcwlldw9/qn7caiow5jhyowchbhhke0ybykkpgwhenypcyaxqdq7tefcrc0dc4k8oiod7qbbrlmzkkcvwze5fcc9hs/we0demha8dztix1iyc2j2tf85kbueuiez/tk4nl0pkrl42qkqwcbwyoyicpuqup7u6iy58eejmkrns0iukkieukgrmv7fiqcoeb6pg2ggfs850ku3ie9xttgd2qqxdimlg/oxlkczh0pavmnxbttzt8r9aozffuswgqxzieedkogzmoq5habjuueyzzuageovw7qtxhhggpje1nmkdpngwnw6ppvm3bo3ltmrejchbalqgae5yagrggdwabdj7bk3ghy+1fe9m+tm+jtkdlpdvgt2jfp7p3oho=</latexit> <latexit sha1_base64="g6o0fzji/vpizil8s528m+cdgfg=">aaacjxicbvbns8naen3ur1q/oh69lbahiprefd0ovl3oryjpq00im+22xbr5yhejlna/48w/4swdrqrp/hu3aq7aora7jzdvmjnnrywkarhfwmfufmfxqbhcwlldw9/qn7caiow5jhyowchbhhke0ybykkpgwhenypcyaxqdq7tefcrc0dc4k8oiod7qbbrlmzkkcvwze5fcc9hs/we0demha8dztix1iyc2j2tf85kbueuiez/tk4nl0pkrl42qkqwcbwyoyicpuqup7u6iy58eejmkrns0iukkieukgrmv7fiqcoeb6pg2ggfs850ku3ie9xttgd2qqxdimlg/oxlkczh0pavmnxbttzt8r9aozffuswgqxzieedkogzmoq5habjuueyzzuageovw7qtxhhggpje1nmkdpngwnw6ppvm3bo3ltmrejchbalqgae5yagrggdwabdj7bk3ghy+1fe9m+tm+jtkdlpdvgt2jfp7p3oho=</latexit> Intuitive Toy Example Suppose structural equation is Y i = X i + 100Z i + 50Z i I(A i = w)+u i So if there are two individuals, one of type w and one of type b, and Z 1 + Z 2 = 1. Without the fairness constraint, type w gets the subsidy even if type b has up to 50 more units of professional ability!

62 Considerations There might be no feasible solution if τ is small enough. It might be the case that the counterfactual gap in each constraint remains constant regardless of Z. These are features of the intervention, not of the fairness framework. Again, a good intervention is a matter of real world design, not of algorithm design!

63 Illustration (Partially Synthetic Data) NYC Public Schools: intervention Z is to provide calculus classes in schools. Attribute A is whether school has a white majority. Outcome Y is proportion of students taking the SAT/ACT. Geographical interference is assumed.

64 Results