AMAS: Attention Model for Attributed Sequence Classification

Transcription

1 <latext sha1_base64="u5xncmv0ktxombzjkhvjvo2ai=">aaab/3cbzdptsjaeman+a/xh+rrsymx8ural3ok8ciro4ajgs7tmug7xazuzuhdqcfwks+gjfj1ufxcxwf+hbwc/z5jdvzjkzxyg508bzvp3sxubw9k55t7k3f3b4vd0+6eg0uxtbowpegyjrs4etg0zhb+lqpkehlvh+hzw7z6h0wvd2ymuhilfjekdhwupcdf1ctexvvlncd/ajqukg1qp70hynehsgcqj1z/ekcxkdkmcp5v+plesoyx9wkkqao8vmpu/fcokm3spv9wrhz9+9ethktj0looxrnq1jp/q/uye90eormymyjoylgucdek7uzf7pappizplbcqml3vpsocdu2nautmywv4nhascn4qzmsq+eq7lu+82qzpfrgc7ghc7bh2toqba0aykmbzak7w5z8678+f8llpltjfzcktyvn4b5aywgg==</latext> <latext sha1_base64="a4gdxlto5jabzev2+erjztv9cjg=">aaab/3cbzbltgjbeizr8ix4ql266uhmxjezrokccmsozwsmjcepgy69prmuntmyisfb3crr3bn3hout+a1bgawav5jj1/+qkpv/0eudau++0utrz3dvek+6wdw6pjk/lpwvvhqwlyyrgivtegggwx2dlccowmcmkucowek7t5vfoesvyppppgn5er5khnffjrydkubuuk27vxyhsgpddbxi1b+wf/jbmaytsmeg17nluyvymksozwfmpn2pmkjvqefysshqh9rpfqtyzz0hcwlnzrk4f6dygk9tqkbgdezvv1+bmf7veasjbp+mysq1ktlwupokymmz/tyzcitaoeyxe2thi2poszydfa2pmliiu5mjzumt57djrrrvc/yvvup/kmnabl3ahtxahrrqhbywgmelvmkb8+y8ox/o57k14oqz57a5+sx50wwgw==</latext> <latext sha1_base64="hs+0g4zhdq59q6az3y7yjjbno18=">aaab/3cbzgbeizr4vgv9slm8yguaozutblwe2wec0dkh0dgqsjj0ztxepeiyspibbpyi7cetrpihxsjpmwt+0pdxvxvv/qdscg1c99spbgxube8ud0t7+wehr+xjk5zouswwyrkrqe5aqoey9wi7ajfdioegoxnezevsjlezj/ggmev2idmmeckatr5k/7pfrrhvdy6ydl4ofcjv6jd/eooeprhghgmqdddzpfezqgxnaqelxqprujamq+xajgme2s/mp07jhxugjeyufbehc/fvreyjrsdrydsjakz6ttyz/6t1uxpe+hmpzwowzotfysqiscjs32taftijjhyou9zestiksqmtwdpsyqhcne8ldlkvuc1qf1vfus37uvwj3pqahnca6x4me1kaodwgcgyg8wcu8oc/ou/phfc5ac04+cwplcr5+aejeloq=</latext> <latext sha1_base64="ci8g7zineyzmemfrpe11gc=">aaab/3cbzgbeizr4vgv9slm8yguaoziugy4cbloybyrb6ojvjk56zprthcemwhscthsgdupuonsbr2elmyrj/apj4q4qq/gmpudau++0uja3tneku6w9/ypdo/lxsusnqwlyzilivcegggwpswm4edrcmkucgwh47tzvf2esvmkfjqtx5ehzepoapgwg+yf90vv9yqoxdzby+hcurq9ms/vuhc0ghjwwtvuuu50vgzvyyzgdsl9uokrvtixytxjrc7wfzu6fkwjodebkvtquft3iqor1pmosj0rso9wpuz/9w6qqlv/yzhmjuys8wmbxejgt2bzlgcpkrewuukw5vjwxefwxgpro0jzvdhtelmwy3moo69c6qnqw791krz5nviqzoidl8oagalchbjsbwrbe4bxengfn3flwphetbsefoyulov+/6newhq==</latext> <latext sha1_base64="zj4uys8vpw7odxdrfwhvojj5tu=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xja3tqu1+s6djlpfomeasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgiv2u4j+6k8cigbwgom682mcvfkagtrmuk27xe1mp6pkccygr/rsdqllqzqarowsrqd72wrvjg8te+awvvzlgyfsx0dgi61hkw+vetxper43jv/rdvmttvozl0lqqllpojav2mr4hcwouajmxmgcyhs3t2l2tbvlxsy/syvbgpgmfdsmmq+h0xw0dwhft+6jfz1kvez7amddiqiokdtdi1uuacx9ia+0df6kx04zafq1kdsp1r4dtfmoxu/xeu4uw==</latext> <latext sha1_base64="e3suspg05b38qeervndokx/7ouk=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xja3tqu1+s6djlpfomeasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgaxq3hnrtwouafkcbrrxap+9igzpbiwqbxuejcx/ywqw5mavjlssudekauhzkgohuz5o9ot60tiddwkvdz6wfx0zjbqerb5vrtq86/nempyv1012opnxcapacmm8juybpjcba44aqyesmlkfpc3orzm1wugrv/zjy0gsayv6xyzd5hbbbq/oewhzrtrxruzlti8o0be6by10tw6qr3e0bv6qt/op/thlj2qu59knvlh2uuz5ez9asxkufq=</latext> <latext sha1_base64="4aybh8rx2pm0qpnvgvrg0n1dlk=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaola6ihhgbeqpultfjnmpvh0nsh1qfex/8rdywdhyycqtgwjlyzae3r13d3p+irg2rvtwchywl5zxyquvtfwzwqtvvwg41qxalyxorrpxoel9a23ah4tbtqybfq8yex437ngztmsbw3owt6er1ihnjgjaw8mt8lilteyarmwounimzu6ulpcot48my74cv+zkvkh0dvpdwmxoukohdoteu/eqzxci3dseb6qajrqutde7amxxcyqbomqzd4amn1floboqv3qphosyir1a10jji9d9bli3x/uwcxayk/ulwrp2ryojkdajylfkjonpdsbk//1uqkjw/2myyq1invojav2mr4hcwouajmxmgcyhs3t2l2rbvlxsy/tsvbgpgmfdsmmq2h3nw0dwft+6jfpriqmy2kv76aardibo0tw6qw3e0cv6rf/ou/tulj2qu/+voqxcs42mytn5ackwufy=</latext> <latext sha1_base64="suskbt71ekg0cgn6xemzyv6ml8m=">aaacyxcbza7t8mwfixd8c6vagmxwqjaag4faijggfgqlqgtvxkodffquetgoqovxafgizaymrblhtbtpyzuth537x1zpbkrg2nvdeczawv1bx1jeqm1vbo7vu3n5hj5l0gajsrtqduilqftubhwlcqgcsdgmrhejfupl6a0t+sdgaxqj+la8ogzaqzlu6wxqmrnjy/lel59bwgr319ffjlpu8alxskxf3aapfbytxotgxhphg4bys+k3frxun3qtwoclqkoybn33sxcmlitbgao1l3paafu2u4e1bue5mgllihhudxsklj0p18srjar9yjczqoe6xbe/fvre5jrudxymmymmc93xub//w6myla/zzldmg2xrrlalsejxofodcatzavltu/yvzmfwxg5j+zjushcmbyvg0yzd6hrdfpnbkr77z6xu2z0tqqou0jag6rxfobt2mlodx2hb/rt+xa2hfzn6jopzw5qdpl1h4bvpe4za==</latext> <latext sha1_base64="e3suspg05b38qeervndokx/7ouk=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xja3tqu1+s6djlpfomeasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgaxq3hnrtwouafkcbrrxap+9igzpbiwqbxuejcx/ywqw5mavjlssudekauhzkgohuz5o9ot60tiddwkvdz6wfx0zjbqerb5vrtq86/nempyv1012opnxcapacmm8juybpjcba44aqyesmlkfpc3orzm1wugrv/zjy0gsayv6xyzd5hbbbq/oewhzrtrxruzlti8o0be6by10tw6qr3e0bv6qt/op/thlj2qu59knvlh2uuz5ez9asxkufq=</latext> <latext sha1_base64="1vmfuq/vvbb/jx6wo0glc55xy=">aaacyxcbza7t8mwfixd8c6vagmxwqjaag4goiyiarec1ibru5zk2x6jr7ycqkd+qn8dmwmgkg26bgbzc2dlrud/1tu6qcq6571xnixfpewv1bxq+sbm1ra7s9vwsayytfgevuuua2cs2gzbgq8pqpohah4dazxo/7jcyje/lghn0ytqxpokmgmv5luugenz8uu5hfdfqvjk99exru6aj0epj0frs5hgcg5bpnr4ryjngjcazj/vpfrxvh3rjwvcclqkoybn33sxsmlitbgao1h3paaxu2u4e1buu5mgllib7uphsklj0l18vljab9yjczqoe6xby/fvre5jrydxymmymmc92xuz//u6mymavzzldmg2wrrlalsejxkfodcataavltu/yvzmfwxg5j+1juv7cmbqvg0yzdahedfuhbor77z6xu2z0sqqox10ag6rxfobt2fmlodx2hb/rt+xdwhfznabopzzzq1pl1h4bwdc4zq==</latext> <latext sha1_base64="suskbt71ekg0cgn6xemzyv6ml8m=">aaacyxcbza7t8mwfixd8c6vagmxwqjaag4faijggfgqlqgtvxkodffquetgoqovxafgizaymrblhtbtpyzuth537x1zpbkrg2nvdeczawv1bx1jeqm1vbo7vu3n5hj5l0gajsrtqduilqftubhwlcqgcsdgmrhejfupl6a0t+sdgaxqj+la8ogzaqzlu6wxqmrnjy/lel59bwgr319ffjlpu8alxskxf3aapfbytxotgxhphg4bys+k3frxun3qtwoclqkoybn33sxcmlitbgao1l3paafu2u4e1bue5mgllihhudxsklj0p18srjar9yjczqoe6xbe/fvre5jrudxymmymmc93xub//w6myla/zzldmg2xrrlalsejxofodcatzavltu/yvzmfwxg5j+zjushcmbyvg0yzd6hrdfpnbkr77z6xu2z0tqqou0jag6rxfobt2mlodx2hb/rt+xa2hfzn6jopzw5qdpl1h4bvpe4za==</latext> <latext sha1_base64="zj4uys8vpw7odxdrfwhvojj5tu=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xja3tqu1+s6djlpfomeasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgiv2u4j+6k8cigbwgom682mcvfkagtrmuk27xe1mp6pkccygr/rsdqllqzqarowsrqd72wrvjg8te+awvvzlgyfsx0dgi61hkw+vetxper43jv/rdvmttvozl0lqqllpojav2mr4hcwouajmxmgcyhs3t2l2tbvlxsy/syvbgpgmfdsmmq+h0xw0dwhft+6jfz1kvez7amddiqiokdtdi1uuacx9ia+0df6kx04zafq1kdsp1r4dtfmoxu/xeu4uw==</latext> <latext sha1_base64="55emaey4r0asqk+uwtj5pqz7rjm=">aaacncbvhltswfjxdv5rjdj2u+hgtsjmigqr0yyzzjo0hwazlskdkmbk+torkwujys3b+a/6g/mfs+8xvml40twuehzopvdhnbukgmvjefeo++tps+cvk+rz7xzvxf112/goskugxflrkkmaduguisr4ubafva40dajfh/3fynv0bpnsg7s0lhedol5bfn1fjkr/+zhxdz2d1luytfvokwyh9+/1lkph/v6vttkxd1sbaidepd53oleksj3nhughiutnttujxa/viuz0fkll/k19vem1vv/gykbi0uvldv/4wdxowxsae1trgffss8pmpwjscazhpsyv3czejjy9clfodc4i+wcxgukhulwtv2/4mcxlpv4saqy2pw+rc3ju/1zpmj+oucyzqzimjuzqjbbk8xqeouqjmxmycyhs3f8vsrrvlx5qzyvlt9ms6phhzmcazgntax+ifxvb2ugvxqe9ral4ghrpfazrei8tqx+ed88fpudw s1yln3qk9cr/9a0f7xqu=</latext> <latext sha1_base64="suskbt71ekg0cgn6xemzyv6ml8m=">aaacyxcbza7t8mwfixd8c6vagmxwqjaag4faijggfgqlqgtvxkodffquetgoqovxafgizaymrblhtbtpyzuth537x1zpbkrg2nvdeczawv1bx1jeqm1vbo7vu3n5hj5l0gajsrtqduilqftubhwlcqgcsdgmrhejfupl6a0t+sdgaxqj+la8ogzaqzlu6wxqmrnjy/lel59bwgr319ffjlpu8alxskxf3aapfbytxotgxhphg4bys+k3frxun3qtwoclqkoybn33sxcmlitbgao1l3paafu2u4e1bue5mgllihhudxsklj0p18srjar9yjczqoe6xbe/fvre5jrudxymmymmc93xub//w6myla/zzldmg2xrrlalsejxofodcatzavltu/yvzmfwxg5j+zjushcmbyvg0yzd6hrdfpnbkr77z6xu2z0tqqou0jag6rxfobt2mlodx2hb/rt+xa2hfzn6jopzw5qdpl1h4bvpe4za==</latext> <latext sha1_base64="zj4uys8vpw7odxdrfwhvojj5tu=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xja3tqu1+s6djlpfomeasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgiv2u4j+6k8cigbwgom682mcvfkagtrmuk27xe1mp6pkccygr/rsdqllqzqarowsrqd72wrvjg8te+awvvzlgyfsx0dgi61hkw+vetxper43jv/rdvmttvozl0lqqllpojav2mr4hcwouajmxmgcyhs3t2l2tbvlxsy/syvbgpgmfdsmmq+h0xw0dwhft+6jfz1kvez7amddiqiokdtdi1uuacx9ia+0df6kx04zafq1kdsp1r4dtfmoxu/xeu4uw==</latext> <latext sha1_base64="e3suspg05b38qeervndokx/7ouk=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xja3tqu1+s6djlpfomeasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgaxq3hnrtwouafkcbrrxap+9igzpbiwqbxuejcx/ywqw5mavjlssudekauhzkgohuz5o9ot60tiddwkvdz6wfx0zjbqerb5vrtq86/nempyv1012opnxcapacmm8juybpjcba44aqyesmlkfpc3orzm1wugrv/zjy0gsayv6xyzd5hbbbq/oewhzrtrxruzlti8o0be6by10tw6qr3e0bv6qt/op/thlj2qu59knvlh2uuz5ez9asxkufq=</latext> AMAS: Attenton Model for Attrbuted Sequence Classfcaton Zhongfang Zhuang Xangnan Kong Elke Rundenstener Abstract Classfcaton over sequental data s mportant for a wde range of applcatons from nformaton retreval, anomaly detecton to genomc analyss. eural network approaches, n partcular recurrent neural networks, have been wdely used n such tasks due to ther strong capablty of feature learnng. However, recent nnovatons n sequence classfcaton learn from not only the sequences but also the assocated attrbutes, called attrbuted sequences. Whle recent work shows the attrbuted sequences to be useful n real-world applcatons, neural attenton models have not yet been explored for attrbuted sequence classfcaton. Ths paper s the frst to study the problem of attrbuted sequence classfcaton wth the neural attenton mechansm. Ths s challengng that now we need to assess the mportance of each tem n each sequence consderng both the sequence tself and the assocated metadata. We propose a framework, called AMAS, to classfy attrbuted sequences usng the nformaton from the sequences, metadata, and the computed attenton. Emprcal results on real-world datasets demonstrate that the proposed AMAS framework sgnfcantly mproves the performance of classfcaton over the state-of-the-art methods on attrbuted sequences. 1 Introducton Classfcaton over sequental data s mportant for a wde range of applcatons over a varety of research areas, ncludng document classfcaton n nformaton retreval and gene classfcaton n genomc analyss. The conventonal approaches to sequence classfcaton focus on only the sequental data, whereas the metadata assocated wth the sequences s often dscarded. However, metadata s has been shown to be useful for a varety of topcs [30, 26]. In the lterature, the data type of a sequence and ts assocated metadata as a unty s called attrbuted sequence. Attrbuted sequences are heterogeneously structured and thus not naturally represented as fxed-sze vectors. For example, genes can be represented as attrbuted sequences, where each gene conssts of a DA sequence and a set of attrbutes {zzhuang, xkong, rundenst}@wp.edu, Computer Scence Department, Worcester Polytechnc Insttute Attrbuted Sequences p1 - p4 <latext sha1_base64="u5xncmv0ktxombzjkhvjvo2ai=">aaab/3cbzdptsjaeman+a/xh+rrsymx8ural3ok8ciro4ajgs7tmug7xazuzuhdqcfwks+gjfj1ufxcxwf+hbwc/z5jdvzjkzxyg508bzvp3sxubw9k55t7k3f3b4vd0+6eg0uxtbowpegyjrs4etg0zhb+lqpkehlvh+hzw7z6h0wvd2ymuhilfjekdhwupcdf1ctexvvlncd/ajqukg1qp70hynehsgcqj1z/ekcxkdkmcp5v+plesoyx9wkkqao8vmpu/fcokm3spv9wrhz9+9ethktj0looxrnq1jp/q/uye90eormymyjoylgucdek7uzf7pappizplbcqml3vpsocdu2nautmywv4nhascn4qzmsq+eq7lu+82qzpfrgc7ghc7bh2toqba0aykmbzak7w5z8678+f8llpltjfzcktyvn4b5aywgg==</latext> <latext sha1_base64="ci8g7zineyzmemfrpe11gc=">aaab/3cbzgbeizr4vgv9slm8yguaoziugy4cbloybyrb6ojvjk56zprthcemwhscthsgdupuonsbr2elmyrj/apj4q4qq/gmpudau++0uja3tneku6w9/ypdo/lxsusnqwlyzilivcegggwpswm4edrcmkucgwh47tzvf2esvmkfjqtx5ehzepoapgwg+yf90vv9yqoxdzby+hcurq9ms/vuhc0ghjwwtvuuu50vgzvyyzgdsl9uokrvtixytxjrc7wfzu6fkwjodebkvtquft3iqor1pmosj0rso9wpuz/9w6qqlv/yzhmjuys8wmbxejgt2bzlgcpkrewuukw5vjwxefwxgpro0jzvdhtelmwy3moo69c6qnqw791krz5nviqzoidl8oagalchbjsbwrbe4bxengfn3flwphetbsefoyulov+/6newhq==</latext> Classfcaton on Attrbutes Classfcaton on Sequences Classfcaton on Attrbuted Sequences P4 can only be <latext sha1_base64="55emaey4r0asqk+uwtj5pqz7rjm=">aaacncbvhltswfjxdv5rjdj2u+hgtsjmigqr0yyzzjo0hwazlskdkmbk+torkwujys3b+a/6g/mfs+8xvml40twuehzopvdhnbukgmvjefeo++tps+cvk+rz7xzvxf112/goskugxflrkkmaduguisr4ubafva40dajfh/3fynv0bpnsg7s0lhedol5bfn1fjkr/+zhxdz2d1luytfvokwyh9+/1lkph/v6vttkxd1sbaidepd53oleksj3nhughiutnttujxa/viuz0fkll/k19vem1vv/gykbi0uvldv/4wdxowxsae1trgffss8pmpwjscazhpsyv3czejjy9clfodc4i+wcxgukhulwtv2/4mcxlpv4saqy2pw+rc3ju/1zpmj+oucyzqzimjuzqjbbk8xqeouqjmxmycyhs3f8vsrrvlx5qzyvlt9ms6phhzmcazgntax+ifxvb2ugvxqe9ral4ghrpfazrei8tqx+ed88fpudw s1yln3qk9cr/9a0f7xqu=</latext> correctly classfed wth attrbuted sequences. Fgure 1: Usng both attrbutes and sequences s essental for predctng the of p 4. e.g., PPI, gene ontology ndcatng the propertes of the gene. etwork traffc can also be modeled as attrbuted sequences. amely, each network transmsson sesson conssts of a sequence of packages beng sent or receved by a router and a set of attrbutes characterzng the network traffc, e.g., date, tme, sze sender prorty etc. Desgnng a classfer to correctly predct the class of an attrbuted sequence s benefcal for applcatons such as automatc network traffc proflng for cyber securty. However, t has been shown that t s challengng to classfy sequences by themselves [31, 24, 19, 25]. Wth dependences between attrbutes and sequences, the attrbuted sequence classfcaton must tackle several mportant desgn challenges. In ths paper, we study the problem of attrbuted sequence classfcaton. Ths problem s dfferent from pror sequence classfcaton work [24, 19] as we now need to extract feature vectors from not only the sequences but also ncorporatng the metadata as the attrbutes, along wth the dependences between attrbutes and sequences. We summarze the challenges below. Attrbute-sequence dependences. Contrary to the smplfyng assumpton that the attrbutes and sequences are ndependent, there are varous dependences between them. Dependences between at- Copyrght c 2019 Copyrght for ths paper s retaned by authors

2 a Classfcaton on data attrbutes [1]. b Sequence classfcaton [31]. Attenton Block learnng the attenton from not only the sequences but also the hybrd of nformaton from both attrbutes and sequences. Our paper offers the followng contrbutons: We formulate the problem of attrbuted sequence classfcaton c Image classfcaton wth attenton [20] logn search search book search d Sequence classfcaton wth attenton [28] logn search search book search e Sequence classfcaton wth attrbute-guded attenton. ths paper Fgure 2: A comparson of related classfcaton problems. trbutes and sequences rase problems when learnng to classfy attrbuted sequence data. For example, n network traffc data, the devce type of the router.e., an attrbutes may affect the pattern of sendng/recevng TCP/UDP packets.e., sequences. Snce conventonal sequence classfcaton approaches focus on a sngle data type only, these dependences would thus not be captured. Attenton from both attrbutes and sequences. Recent sequence learnng research [34] has used neural attenton models to mprove the performance of sequence learnng, such as document classfcaton [34]. However, the attenton mechansm focuses on learnng the weght of certan tme steps or sub-sequences n each sequental nstance, wthout regards to ts assocated attrbutes. Wth nformaton from the attrbutes, the weght of tem or sub-sequence may be drastcally dfferent from the weght calculated by the attenton mechansm usng only sequences, whch would consequently have dfferent classfcaton results. To address the above challenges, we propose an endto-end soluton for attrbuted sequence classfcaton wth an attenton model, called AMAS. The AMAS model ncludes three man components: a Attrbute et to encode the nformaton from attrbutes, a Sequence et to encode the nformaton from sequences, and an We desgn a deep learnng framework, called AMAS, wth two attenton-based models to explot the nformaton from attrbutes and sequences. We demonstrate that the proposed models sgnfcantly mprove the performance of attrbuted sequence classfcaton usng performance experments and case studes. 2 Problem Formulaton 2.1 Prelmnares. We ntroduce the key defntons n ths secton. The mportant notatons are summarzed n Table 1. Defnton 1. Sequence Gven a fnte set I composed of r categorcal tems, a sequence s = x 1,, x t s an ordered lst of t tems, where x t I. We use the subscrpt to dstngush dfferent sequence nstances. Followng common pre-processng steps for deep learnng, we apply zero-paddng on the sequences of varable-length, n whch each sequence s padded to the maxmum length of the sequences n a dataset usng 0 s. The second step s to one-hot encode each sequence s [10]. We denote the [ one-hot encoded ] form of sequence s as a matrx s = x 1,, x t. Learnng models are capable of dsregardng the paddng so that the paddng has no effect on the tranng of models. Defnton 2. Attrbuted Sequence Gven a u- dmensonal attrbute vector v s composed of u attrbutes, an attrbuted sequence p s a par composed of an attrbute vector v and a one-hot encoded sequence s, denoted as p = v, s. 2.2 Problem Defnton. We formulate the attrbuted sequence classfcaton problem as the problem of fndng the parameters θ of a predctor Θ that mnmzes the predcton error of class s. Intutvely, we want to maxmze the possblty of correctly predctng s when gven a tranng set P = {p 1,, p k } of k attrbuted sequences. Thus, we formulate the tranng process as an optmzaton process: 2.1 arg mn θ Prl log Pr Θ p Copyrght c 2019 Copyrght for ths paper s retaned by authors

3 Table 1: Important Mathematcal otatons otaton Descrpton R The set of real numbers. P A set of attrbuted sequences. r The number of all possble tems n sequences. s A sequence of categorcal tems. x t The t-th tem n sequence s. t max The maxmum length of sequences. s A one-hot encoded sequence n the form of a matrx s R t max r. x t A one-hot encoded tem at t-th tme step. v An attrbute vector. p An attrbuted sequence..e.,, p = v, s p A feature vector of attrbuted sequence p. µ Attenton weghts. α Attenton vector. Our goal s to fnd the parameters that mnmze the categorcal cross-entropy loss between the predcted s usng parameters n functon Θ and the true s for all attrbuted sequences n the dataset. 3 Attrbuted Sequence Attenton Mechansm The proposed AMAS model has three components, one Attrbute et for learnng the attrbute nformaton, one Sequence et to learn the sequental nformaton, and one Attenton Block to learn the attenton from both attrbutes and sequences. 3.1 etwork Components Attrbute et. We buld Attrbute et usng fully connected neural network denoted as: 3.2 f A; W r, b r = tanhw r A + b r where W a and b a are two tranable parameters n the Attrbute et, denotng the weght matrx and bas vector, respectvely. We use the actvaton functon tanh here n our Attrbute et based on our emprcal studes. Other choces, such as ReLu or sgmod, may work equally well n other real-world scenaros. When gven an attrbuted sequence p = v, s, Attrbute et takes the attrbutes as nput and generates an attrbute vector r = fv ; W r, b r. Dfferent from prevous work n [4, 27] usng stacked fully connected neural network as autoencoder, where the tranng goal s to mnmze the reconstructon error, our goal of Attrbute et s to work together wth other network components to maxmze the possblty of predctng the correct s Sequence et. Dfferent from the attrbutes beng unordered, tems n our sequences have a temporal orderng. The nformaton about the sequences s n both the tem values and the orderng of tems. The temporal orderngs requre a model that s capable of handlng the dependences between dfferent tems. There have been extensve studes on usng recurrent neural networks R to handle temporal dependences. However, R suffers from the problem of explodng and vanshng gradent durng the tranng, where the gradent value becomes too large or too small and thus the network becomes untranable. Long Short- Term Memory LSTM [13] s desgned as one varaton and expanson of the R to handle such ssues. LSTM s capable of rememberng values over long tme ntervals by ntroducng addtonal nternal varables.e., varous gates and cell states. We use an LSTM to handle the dependences. Wth a varable X t at tme t, the Sequence et can be expressed as: 3.3 t = σ W X t + U h t 1 + b f t = σ W f X t + U f h t 1 + b f o t = σ W o X t + U o h t 1 + b o g t = tanh W c X t + U c h t 1 + b c c t = f t c t 1 + t g t h t = o t tanh c t where denotes the btwse multplcaton, σ s a sgmod actvaton functon, t, f t and o t are the nternal gates of the LSTM, and c t and h t are the cell and hdden states of the LSTM, respectvely. We denote the Sequence et as: 3.4 gx t ; W s, U s, b s = h t where W s = [W, W f, W o, W c ], U s = [U, U f, U o, U c ] and b s = [b, b f, b o, b c ]. [ ] Wth the sequence s = x 1,, x t as part of an attrbuted sequence p, the hdden states for nput x t are g x t ; W s, U s, b s = h t Attenton Block. Recent work [20, 6] has dentfed that even LSTM-based solutons cannot fully handle the sequence learnng on long sequences that the nformaton over a long tme may be lost. One popular soluton to ths problem s to ncorporate the attenton mechansm nto the model. The attenton mechansm effectvely summarzes the data wth the am to leverage the mportance of each tem n the sequental nput. Attrbuted Sequence Attenton. Dfferent from the common sequence attenton models, we now need to ncorporate the attrbute nformaton nto Copyrght c 2019 Copyrght for ths paper s retaned by authors

4 <latext sha1_base64="+newlwycyydy8+le/yujfrp7bbk=">aaacbxcbzc7sgbfiznvcz41ralabbkuzaabmwsrnbxcs7hnj2wtizowwfyesqx0aw30eo7h1oxwcx8jsovj/ghg4z/ncm78wrumyd49jy2t7z3dkt75f2dw6pjyslpw2dwewrjgwqm2ajapogwoydkuczbmgnwr86t3nkbpmokhm5eqczwukueg2c9rpinebd92q9ug1owl78oyqfvvkjzr/xeg4xydsiqhrxuhye0cy6voytbtbxzdrktmr5cz6hahhsczw+e+pfogfhpptwtxp+7fydyzlwe8mr1cmxgeru2m/+r9axjb+occmkcljylfrmm8yf/d4fuaxesikdtbr1t/pkhbumxmw0tmxkoqiyt8sumxa1h3vox9dcx/dbtd4omqhc3sbrlcibladvattrbbhl2gv/tmpxvv3of3uwjd8iqzm7qk7+sxbhszxw==</latext> <latext sha1_base64="2ydsze4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1sshm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetp6fkccbajs40pjq6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx544p1hvz1yry8o6jxxde7bjdersm9tebokiedelaqb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="gtmv6ivthka+4horls3tcoxpqlm=">aaacfhcbvdlssafj34rpuvfdy4gsxcxvgsxey4kblcvybbqyt6u07dpjgzkum/wa9zqj7gtt+79an/dszufbt1w4xdovdzd8wloplksb2ldw19y7o0vd7e2d3bw8o2zjkbiuwjxgkuh6rwfkilcuuh24sgaqeh443vs39zmiyalwxk1caiydjnpkffacs3jfkduypptueayh7sqxhzxdz65zswqwvpgzwixpiikf3zpz+iabjaqcgnuvzsk1zosorlew7csyklhzag9tumsghtsaf4mn2llgp1i6akvnqp/l1issdkjpl2zp5wlx7+5/us5d84kqvjrefiz4/8hgmv4bwmpgacqoittqgvtgffdeqeoupxvclyccyjyvdtp2yg/lph1zszw/syr1rtfrcz2gu1rfrpgddratdrcfd2hf/sk3oxn4934md5nqytgcxoe5mb8/qigqz6q</latext> <latext sha1_base64="xg38qqc07e6ma225thsezgyyqsy=">aaab/ncbzdlsgmxfiyzxmu9jyor8e1ivlrhe6lljpsok9qdsomttthazkgsemgz4km5ckolw53dn252s9dwhwif/zmhc/jhklftpo/bwvldw9/ylg2vt3d29/bdg8o2tlkfsqsnlfhdcgncqcatqw0jxaki4hejnwh8o613honsbh3zjjwfq0jhzkwvusd9yam5f9ykomppacxp3novvxat5mcbn8aqgud0v/qdbkeccimz0rrne9iegvkgykbycj/vrci8rkpssygqjzrizufn8mw6axgnyj5h4mz9pzehrvwer7atizps7wp+v+tl5r4jsokkkhas8xxsmdjohtlocakoim1hawff7k8qjpba2rgydcff/piytc9rvuu7r1jvfhguwak4bvxgg2tqbw3qbc2aqqaewst4c56cf+fd+z3rjjfzbh4i+fzb/fhlm0=</latext> <latext sha1_base64="2ydsze4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1sshm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetp6fkccbajs40pjq6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx544p1hvz1yry8o6jxxde7bjdersm9tebokiedelaqb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="ke/ptvz4zdrkas+upaaqfp1c+2q=">aaacfhcbvdlssafj34rpuvfdy4crahlxj7osuomygn1ag8ketmonuzczeqomb/hb7jvt3anbt37bf6gkzyl23rgwugce7mh48emsmxb38ba+sbm1nzpp7y7t39wab4dd2sucajterfi9hwsgveobuuvg14saic+g64/uc397miss+r6yxuceecrpqgpwwpp0egi19o0nhn0ia0qj17vlzpprg1ewzrltgfqaaclc/8gqwjkotafwfyyr5jx8psvcummjkg0rcjmkej6cvkcchsded5c+sc60mrsaseryzurfxshuk5dx2/maewyl4v/ef1ebtdusnmckobk/himkuky/dglibrlgpjpgiqraziwfjkpxtvaluccyjkvdtpocg+rpfovozrf2zvgs+ohm7qoaob12jbmqfmojgp7qc3pfb8az8w58gj/z1twjudlbczc+fgefdp6p</latext> <latext sha1_base64="l7u7aa1n2mxz5g7ot1dxzmoeikg=">aaab/ncbzdlsgmxfiyzxmu9jyor8e1ivlphtdftx0wcfeob2htjppq5msdjcgqz8ftcufhhrc7jzbuzbwwjrd4gp/5zdofkjyag2nvftrk1vbg5tl3bku3v7b4fu0xfhj6ncpi0tlqhehdrhvjc2oyarnlqe8yrbjs5ndw7j0rpmoh7m5uk4ggkaewxmtyk3dob5fem85a+zfutungf65d56fa8mjcxxaw/gaoo1ardr8ewwsknwmcgto77njrbhpshmjg8peg1kqhp0ij0lqreq6y+fk5vldoemajsk8yohd/t2siaz3lke3kyiz1cm1m/lfrpya+ctiqzgqiwitfccqgseasczkmddphyqvttecveykystaxsq/cxv7wknxrt3znvrrio4pnoap8ca0aoalaoa0wymazeavvzppz4rw7h4vwaeyoqf/5hz+ao/blmw=</latext> <latext sha1_base64="2ydsze4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1sshm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetp6fkccbajs40pjq6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx544p1hvz1yry8o6jxxde7bjdersm9tebokiedelaqb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="ppmkzzdoerbr3fzpsrauq4anq2e=">aaacf3cbzc7sgbfiznvcz4w7usm8egxmkwq4wwazuuecwfknwznzwkq2z3h7kiyqk+hw9gq49gj7awpogv4ersmmqfbj7+cw7nzb8jzpt2vg9nzxvtfwmzt5xf3tnd23cpdusqzjcjay8lc2ikoasgzpmmktscbxxkerdw7h9cyjsmxs5f4pbqqx6swsyyjr1grd47aii9w2gndrjyf7yio6zbdx56pqlxglbyk8dp4mcmmauj+tdsptekmnkvmv3ha4yijwjheb5tlegcb2qhrqsjqgfwstl4zwmxu6ujtk+xkj+7fyzesg3jyhbgrpfvym1s/ldrgd29ctkwckmhodfxcoxtve4d9xheqjmqwuesmzvxbrpjkhapja3xyebbm8jyzfzghzahflnzld16hxjlllemn6bqvky+uurlvubxveevp6aw9ojfn2xl3ppzpaeukm5s5qnyvn4b3lufna==</latext> <latext sha1_base64="h6mcvzrsgm5upml8x+0mrweya=">aaacf3cbzc7sgbfiznvcz4w7usm8egxmkwm0blge3kcoycsvxmjyfjkndys5cwbafwwew1uewe1tln8dxchiptoipax//oydz5g8fz0p73reztr6xubwd28nv7u0fhlphxw2vgemhthoeyfzifhawq10zzaeljjao5amr7etevmrpgjjfk/harorgcsszyjr1grc046iqtwxgnaxjaf7sis6yfflyyxwc17jmwqvgj+hapqrfrg/nv5ctqsxppwo1fy9obspkzprdlm+yxqiqkdkag2lmyladdppfzj8yz0e7fsvljjqft3iwruumotj0r0uo1xjuy/9xarvdvumlhdeq09mvufyj3sb+4xcvtzsqvcjbo3yjokklbtu1vyysraaoyyve3gx85hfrrlkm/5ztuqvomcugmnam8te1qqaqqqe6ougjvabx9oy8o+/oh/m5a11z5jmnaeho1y/cwj+b</latext> <latext sha1_base64="sgwewobqlwop/xciuubipjmyn+4=">aaacf3cbzdlsgmxfiyz9vbrrepk3aslubewgte6lljpsok9qkcomfs0du1mqpiryjd4hd6aw30ed+lwpu/ga5hefrb1h8dhf87hnpyh5ewb1/12cmvrg5tb+e3czu7e/khx8kp40rracyx6odeg2crdawzhboswvehbxa4eh2um89gtisju7wejxkehe+owsy62geojlewi/kytliqnyq1o2abv0lrkgwhir7lr4fbw5lbc9ad44/dmgidove647nsttkmcsgkfqjbejoa+hyjiga3u2nx8jwuxv6ub8r+ykdp+7fzqircctj2cmkferk3m/2qdxprvumlzgigorf/yrje+jhrjhffddxxyivczeumqkektw1hsyihcmcufwwy3nioq9c8qnw79xsttbpki9o0rkqiw9doyqqotpqiiqe0at6rw/os/pufdfs9acm585rgtyvn4b2ywfmg==</latext> <latext sha1_base64="mnwra8armv44xzwgxstrea/fm=">aaacfxcbzc7tsmwfiadcvlfmbgylgokmpsjswwvmlpwcr6kdoqoe5ja9vjltbqqi8bw/aco/ahlzeqjea7fqft+ydkn/5yjc/whgjolhefbkm1sbm3vlhcre/shh0f28ulhjamk0kyjt2qviao46gtmebqexjifhdobpo7wb37bfkxjh7quwfereyxcxkl2l+ftyquyahqsbcjinphroa9tlv7ttvp+7mhdfblackcrv8+2cwtggaqawpj0r1xudolysm8ohrwxsbylqcrlb32bmilbev9aj+m8rhis2lz67fycyek1jqltgre9vqu1mflfrz/q8blwcxsdtfdlaptjnwcz2ngizaz8aifqycyumyyij1sazps2pgemasv4xybroaxd57rugr53qo1mkveznamlvemuuke1eqt1eyu5egfvai369l6tz6sz0vryspmttgsrk9f6wefka==</latext> <latext sha1_base64="fihe3xjkd6ju2rqmqg1cqc+qw=">aaacfhcbvdlssafj3uv62vqodgtbaidwfj3oy4kblcvybbqyt6u07ddijmxohxpygh+bwp8gduhxvf/gbttosboubc4dz7uuejh8zkpvtfxultfwza3ydmvnd2//wdw86sgoeqtajgkr6plyaqmc2ooqbr1yaa59bl1/cpv73ucqkkb8xk1jcem84jsgbcsteebjimrq7afpoppoq1pthr10ljlprp1ewzrltgfqaiclc/8gqwjkotafwfyyr5jx8psvcummgqg0rcjmkej6cvkcchsded5c+sc60mrsaseryzurfxshuk5dx2/maewyl4v/ef1ebtdusnmckobk/himkuky/dglibrlgpjpgiqraziwfjkpxtvaluccyjjvddpocg+rphvdzs/s6uztfrgz2m1rddrpgddreldrgbd2hf/sk3oxn4934md7nqywjudlgczc+fgedc56o</latext> <latext sha1_base64="vbafdy+0zk7w+2mnb7mw66wfo=">aaab/ncbzdlsgmxfiyz9vbrbvrcuqkwos4sm250wxdtzqv7gxycmmmmdu0yickizrjwvdy4umstz+hotzftz6gtpwq+/nmo5+spjkpae63u1pb39jckm9xdnb39g/cw6ootlkfsrsnlfg9cgncqcbtqw0jpaki4hej3whyo6t3h4nsbh3zpjwfi0jhzkwvucdyam5f9ygomppas+hd56fa9ujcxxaw/gcoo1ardr8ewwsknwmcgto77njrbhpshmjg8mkg1kqhp0ij0lqreq6y+fk5plfoemajsk8yohd/t2siaz3lke3kyiz1cm1m/lfrpya+ctiqzgqiwitfccqgseasczkmddphyqvttecveykystaxq/cxv7wknau6b/noqzaarrxlcaroqa344bo0qbo0qbtgkifn8arencfnxxl3phatjaeyoqz/5hz+ao5vlms=</latext> <latext sha1_base64="2ydsze4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1sshm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetp6fkccbajs40pjq6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx544p1hvz1yry8o6jxxde7bjdersm9tebokiedelaqb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="4stvxtr8vdpbr6fgmjp1ow2tlui=">aaacencbvdlssafj34rpuvhzs3g0wom5k40wxbtzcv7apagbtm3bozbjmjki+qs/wk1+gjtx6w/4bf6gkzyl23rgwugce7mh48ecke0439ba+sbm1nzpp7y7t39wab8dt1wusaotgvfidn2gdmblc00h24sgyq+h44/uc39zixsjxr6cxdeiyexglggjefzppyr67afpoppyq1rvhrvmplv1jwz8cpxc1jbbzqe/dmfrjqjqwjkvi914n1icvsm8ohk/ctbtghezkcnqgchkag6sx9h+mmsrbjm0ijwfq34uuhepq99s5lnvspel/3m9rac3g5sjoeg6pxrkhcsi5xxgydmatv8aghkpmsmi6jjfsbwha+jpfiakyysmngxe5hlbsvaq7hd06l3g6kqezdi6qyexxqi4aqilakin9ije0zv1bl1bh9bnfhxkm5o0aksr18taz42</latext> <latext sha1_base64="adkmcmxvsu2los2dzqtmth55m=">aaab+ncbzc7tsmwfiadcvllsliylehlavkwgcsxkxspqtsfyxke1ajuw7ycqkedhyqahvp6ejbfbbtayy9z+vsfc3so/0gyqo3nftuljc2t7z3ybmvv/+dwyk0ed3wskkw6oggj6kdie0yf6rhqgollrrcpgolf05t5vfdalkajudmzsqkoxolgfcjrdctdwpmpmh9d6rm5be5kfb8xreqnad/ajqofa7dl+gowsnnadgdj64hvsbblshmjg8sow1uqpevjmraoecc6yban5/dcomyj8o+yedc/t2ria71jee2kymz0au1uflfbzca+driqjcpiqivf8upgyab8xzgcqcdztzqfhreyvee6qqjatg3bx/3yonqvg77lw6/wbbvxlmepoa14imr0aqt0aydgmejeaav4m15cl6cd+dj2vpypkt8efo5w9qajqf</latext> <latext sha1_base64="+newlwycyydy8+le/yujfrp7bbk=">aaacbxcbzc7sgbfiznvcz41ralabbkuzaabmwsrnbxcs7hnj2wtizowwfyesqx0aw30eo7h1oxwcx8jsovj/ghg4z/ncm78wrumyd49jy2t7z3dkt75f2dw6pjyslpw2dwewrjgwqm2ajapogwoydkuczbmgnwr86t3nkbpmokhm5eqczwukueg2c9rpinebd92q9ug1owl78oyqfvvkjzr/xeg4xydsiqhrxuhye0cy6voytbtbxzdrktmr5cz6hahhsczw+e+pfogfhpptwtxp+7fydyzlwe8mr1cmxgeru2m/+r9axjb+occmkcljylfrmm8yf/d4fuaxesikdtbr1t/pkhbumxmw0tmxkoqiyt8sumxa1h3vox9dcx/dbtd4omqhc3sbrlcibladvattrbbhl2gv/tmpxvv3of3uwjd8iqzm7qk7+sxbhszxw==</latext> <latext sha1_base64="2ydsze4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1sshm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetp6fkccbajs40pjq6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx544p1hvz1yry8o6jxxde7bjdersm9tebokiedelaqb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="ppmkzzdoerbr3fzpsrauq4anq2e=">aaacf3cbzc7sgbfiznvcz4w7usm8egxmkwq4wwazuuecwfknwznzwkq2z3h7kiyqk+hw9gq49gj7awpogv4ersmmqfbj7+cw7nzb8jzpt2vg9nzxvtfwmzt5xf3tnd23cpdusqzjcjay8lc2ikoasgzpmmktscbxxkerdw7h9cyjsmxs5f4pbqqx6swsyyjr1grd47aii9w2gndrjyf7yio6zbdx56pqlxglbyk8dp4mcmmauj+tdsptekmnkvmv3ha4yijwjheb5tlegcb2qhrqsjqgfwstl4zwmxu6ujtk+xkj+7fyzesg3jyhbgrpfvym1s/ldrgd29ctkwckmhodfxcoxtve4d9xheqjmqwuesmzvxbrpjkhapja3xyebbm8jyzfzghzahflnzld16hxjlllemn6bqvky+uurlvubxveevp6aw9ojfn2xl3ppzpaeukm5s5qnyvn4b3lufna==</latext> <latext sha1_base64="xg38qqc07e6ma225thsezgyyqsy=">aaab/ncbzdlsgmxfiyzxmu9jyor8e1ivlrhe6lljpsok9qdsomttthazkgsemgz4km5ckolw53dn252s9dwhwif/zmhc/jhklftpo/bwvldw9/ylg2vt3d29/bdg8o2tlkfsqsnlfhdcgncqcatqw0jxaki4hejnwh8o613honsbh3zjjwfq0jhzkwvusd9yam5f9ykomppacxp3novvxat5mcbn8aqgud0v/qdbkeccimz0rrne9iegvkgykbycj/vrci8rkpssygqjzrizufn8mw6axgnyj5h4mz9pzehrvwer7atizps7wp+v+tl5r4jsokkkhas8xxsmdjohtlocakoim1hawff7k8qjpba2rgydcff/piytc9rvuu7r1jvfhguwak4bvxgg2tqbw3qbc2aqqaewst4c56cf+fd+z3rjjfzbh4i+fzb/fhlm0=</latext> <latext sha1_base64="gtmv6ivthka+4horls3tcoxpqlm=">aaacfhcbvdlssafj34rpuvfdy4gsxcxvgsxey4kblcvybbqyt6u07dpjgzkum/wa9zqj7gtt+79an/dszufbt1w4xdovdzd8wloplksb2ldw19y7o0vd7e2d3bw8o2zjkbiuwjxgkuh6rwfkilcuuh24sgaqeh443vs39zmiyalwxk1caiydjnpkffacs3jfkduypptueayh7sqxhzxdz65zswqwvpgzwixpiikf3zpz+iabjaqcgnuvzsk1zosorlew7csyklhzag9tumsghtsaf4mn2llgp1i6akvnqp/l1issdkjpl2zp5wlx7+5/us5d84kqvjrefiz4/8hgmv4bwmpgacqoittqgvtgffdeqeoupxvclyccyjyvdtp2yg/lph1zszw/syr1rtfrcz2gu1rfrpgddratdrcfd2hf/sk3oxn4934md5nqytgcxoe5mb8/qigqz6q</latext> <latext sha1_base64="2ydsze4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1sshm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetp6fkccbajs40pjq6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx544p1hvz1yry8o6jxxde7bjdersm9tebokiedelaqb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="h6mcvzrsgm5upml8x+0mrweya=">aaacf3cbzc7sgbfiznvcz4w7usm8egxmkwm0blge3kcoycsvxmjyfjkndys5cwbafwwew1uewe1tln8dxchiptoipax//oydz5g8fz0p73reztr6xubwd28nv7u0fhlphxw2vgemhthoeyfzifhawq10zzaeljjao5amr7etevmrpgjjfk/harorgcsszyjr1grc046iqtwxgnaxjaf7sis6yfflyyxwc17jmwqvgj+hapqrfrg/nv5ctqsxppwo1fy9obspkzprdlm+yxqiqkdkag2lmyladdppfzj8yz0e7fsvljjqft3iwruumotj0r0uo1xjuy/9xarvdvumlhdeq09mvufyj3sb+4xcvtzsqvcjbo3yjokklbtu1vyysraaoyyve3gx85hfrrlkm/5ztuqvomcugmnam8te1qqaqqqe6ougjvabx9oy8o+/oh/m5a11z5jmnaeho1y/cwj+b</latext> <latext sha1_base64="l7u7aa1n2mxz5g7ot1dxzmoeikg=">aaab/ncbzdlsgmxfiyzxmu9jyor8e1ivlphtdftx0wcfeob2htjppq5msdjcgqz8ftcufhhrc7jzbuzbwwjrd4gp/5zdofkjyag2nvftrk1vbg5tl3bku3v7b4fu0xfhj6ncpi0tlqhehdrhvjc2oyarnlqe8yrbjs5ndw7j0rpmoh7m5uk4ggkaewxmtyk3dob5fem85a+zfutungf65d56fa8mjcxxaw/gaoo1ardr8ewwsknwmcgto77njrbhpshmjg8peg1kqhp0ij0lqreq6y+fk5vldoemajsk8yohd/t2siaz3lke3kyiz1cm1m/lfrpya+ctiqzgqiwitfccqgseasczkmddphyqvttecveykystaxsq/cxv7wknxrt3znvrrio4pnoap8ca0aoalaoa0wymazeavvzppz4rw7h4vwaeyoqf/5hz+ao/blmw=</latext> <latext sha1_base64="ke/ptvz4zdrkas+upaaqfp1c+2q=">aaacfhcbvdlssafj34rpuvfdy4crahlxj7osuomygn1ag8ketmonuzczeqomb/hb7jvt3anbt37bf6gkzyl23rgwugce7mh48emsmxb38ba+sbm1nzpp7y7t39wab4dd2sucajterfi9hwsgveobuuvg14saic+g64/uc397miss+r6yxuceecrpqgpwwpp0egi19o0nhn0ia0qj17vlzpprg1ewzrltgfqaaclc/8gqwjkotafwfyyr5jx8psvcummjkg0rcjmkej6cvkcchsded5c+sc60mrsaseryzurfxshuk5dx2/maewyl4v/ef1ebtdusnmckobk/himkuky/dglibrlgpjpgiqraziwfjkpxtvaluccyjkvdtpocg+rpfovozrf2zvgs+ohm7qoaob12jbmqfmojgp7qc3pfb8az8w58gj/z1twjudlbczc+fgefdp6p</latext> <latext sha1_base64="2ydsze4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1sshm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetp6fkccbajs40pjq6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx544p1hvz1yry8o6jxxde7bjdersm9tebokiedelaqb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="sgwewobqlwop/xciuubipjmyn+4=">aaacf3cbzdlsgmxfiyz9vbrrepk3aslubewgte6lljpsok9qkcomfs0du1mqpiryjd4hd6aw30ed+lwpu/ga5hefrb1h8dhf87hnpyh5ewb1/12cmvrg5tb+e3czu7e/khx8kp40rracyx6odeg2crdawzhboswvehbxa4eh2um89gtisju7wejxkehe+owsy62geojlewi/kytliqnyq1o2abv0lrkgwhir7lr4fbw5lbc9ad44/dmgidove647nsttkmcsgkfqjbejoa+hyjiga3u2nx8jwuxv6ub8r+ykdp+7fzqircctj2cmkferk3m/2qdxprvumlzgigorf/yrje+jhrjhffddxxyivczeumqkektw1hsyihcmcufwwy3nioq9c8qnw79xsttbpki9o0rkqiw9doyqqotpqiiqe0at6rw/os/pufdfs9acm585rgtyvn4b2ywfmg==</latext> <latext sha1_base64="vbafdy+0zk7w+2mnb7mw66wfo=">aaab/ncbzdlsgmxfiyz9vbrbvrcuqkwos4sm250wxdtzqv7gxycmmmmdu0yickizrjwvdy4umstz+hotzftz6gtpwq+/nmo5+spjkpae63u1pb39jckm9xdnb39g/cw6ootlkfsrsnlfg9cgncqcbtqw0jpaki4hej3whyo6t3h4nsbh3zpjwfi0jhzkwvucdyam5f9ygomppas+hd56fa9ujcxxaw/gcoo1ardr8ewwsknwmcgto77njrbhpshmjg8mkg1kqhp0ij0lqreq6y+fk5plfoemajsk8yohd/t2siaz3lke3kyiz1cm1m/lfrpya+ctiqzgqiwitfccqgseasczkmddphyqvttecveykystaxq/cxv7wknau6b/noqzaarrxlcaroqa344bo0qbo0qbtgkifn8arencfnxxl3phatjaeyoqz/5hz+ao5vlms=</latext> <latext sha1_base64="fihe3xjkd6ju2rqmqg1cqc+qw=">aaacfhcbvdlssafj3uv62vqodgtbaidwfj3oy4kblcvybbqyt6u07ddijmxohxpygh+bwp8gduhxvf/gbttosboubc4dz7uuejh8zkpvtfxultfwza3ydmvnd2//wdw86sgoeqtajgkr6plyaqmc2ooqbr1yaa59bl1/cpv73ucqkkb8xk1jcem84jsgbcsteebjimrq7afpoppoq1pthr10ljlprp1ewzrltgfqaiclc/8gqwjkotafwfyyr5jx8psvcummgqg0rcjmkej6cvkcchsded5c+sc60mrsaseryzurfxshuk5dx2/maewyl4v/ef1ebtdusnmckobk/himkuky/dglibrlgpjpgiqraziwfjkpxtvaluccyjjvddpocg+rphvdzs/s6uztfrgz2m1rddrpgddreldrgbd2hf/sk3oxn4934md7nqywjudlgczc+fgedc56o</latext> <latext sha1_base64="2ydsze4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1sshm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetp6fkccbajs40pjq6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx544p1hvz1yry8o6jxxde7bjdersm9tebokiedelaqb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="mnwra8armv44xzwgxstrea/fm=">aaacfxcbzc7tsmwfiadcvlfmbgylgokmpsjswwvmlpwcr6kdoqoe5ja9vjltbqqi8bw/aco/ahlzeqjea7fqft+ydkn/5yjc/whgjolhefbkm1sbm3vlhcre/shh0f28ulhjamk0kyjt2qviao46gtmebqexjifhdobpo7wb37bfkxjh7quwfereyxcxkl2l+ftyquyahqsbcjinphroa9tlv7ttvp+7mhdfblackcrv8+2cwtggaqawpj0r1xudolysm8ohrwxsbylqcrlb32bmilbev9aj+m8rhis2lz67fycyek1jqltgre9vqu1mflfrz/q8blwcxsdtfdlaptjnwcz2ngizaz8aifqycyumyyij1sazps2pgemasv4xybroaxd57rugr53qo1mkveznamlvemuuke1eqt1eyu5egfvai369l6tz6sz0vryspmttgsrk9f6wefka==</latext> <latext sha1_base64="adkmcmxvsu2los2dzqtmth55m=">aaab+ncbzc7tsmwfiadcvllsliylehlavkwgcsxkxspqtsfyxke1ajuw7ycqkedhyqahvp6ejbfbbtayy9z+vsfc3so/0gyqo3nftuljc2t7z3ybmvv/+dwyk0ed3wskkw6oggj6kdie0yf6rhqgollrrcpgolf05t5vfdalkajudmzsqkoxolgfcjrdctdwpmpmh9d6rm5be5kfb8xreqnad/ajqofa7dl+gowsnnadgdj64hvsbblshmjg8sow1uqpevjmraoecc6yban5/dcomyj8o+yedc/t2ria71jee2kymz0au1uflfbzca+driqjcpiqivf8upgyab8xzgcqcdztzqfhreyvee6qqjatg3bx/3yonqvg77lw6/wbbvxlmepoa14imr0aqt0aydgmejeaav4m15cl6cd+dj2vpypkt8efo5w9qajqf</latext> <latext sha1_base64="4stvxtr8vdpbr6fgmjp1ow2tlui=">aaacencbvdlssafj34rpuvhzs3g0wom5k40wxbtzcv7apagbtm3bozbjmjki+qs/wk1+gjtx6w/4bf6gkzyl23rgwugce7mh48ecke0439ba+sbm1nzpp7y7t39wab8dt1wusaotgvfidn2gdmblc00h24sgyq+h44/uc39zixsjxr6cxdeiyexglggjefzppyr67afpoppyq1rvhrvmplv1jwz8cpxc1jbbzqe/dmfrjqjqwjkvi914n1icvsm8ohk/ctbtghezkcnqgchkag6sx9h+mmsrbjm0ijwfq34uuhepq99s5lnvspel/3m9rac3g5sjoeg6pxrkhcsi5xxgydmatv8aghkpmsmi6jjfsbwha+jpfiakyysmngxe5hlbsvaq7hd06l3g6kqezdi6qyexxqi4aqilakin9ije0zv1bl1bh9bnfhxkm5o0aksr18taz42</latext> a Attrbuted Sequence Attenton. b Attrbuted Sequence Hybrd Attenton. Fgure 3: Two types of attenton [ for attrbuted ] sequence classfcaton n ths paper, where µ = µ 1,, µ t s the attenton weghts. the learnng process. Here, we desgn the Attenton Block as follows: Frst, we need to compute the attenton weght µ t at t-th tme as: g x t = h t exp g x t µ t = t j=1 g x j Then, the attenton weght s multpled wth the hdden state at each tme step: 3.5 α t = µ t g x t, t = 1, 2,, t The attenton weght µ t at t tme s randomly ntalzed and ncrementally adjusted durng the tranng process. The output at each tme step s then augmented wth the outputs from Attrbute et as: p t = fv α t At the last tme step t, we denote p = p t to smply the notaton. Attrbuted Sequence Hybrd Attenton. Dfferent from the prevous approach, the outputs of Attrbute et and Sequence et are augmented wth the Attenton Block. The attenton weght s wrtten as: d v, x t µ t = = fv g d exp t j=1 d v, x t x t v, x j Then, the vectors used for classfcaton s: 3.6 α t = µ t d v, x t, t = 1, 2,, t 3.2 Attrbuted Sequence Classfcaton. In the soluton of attrbuted sequence classfcaton wthout attenton, the Attrbute et and the Sequence et are frst concatenated as: 3.7 p = d v, x t = r h t Here, denotes the concatenaton and t denotes the last tem n s. Although all attrbuted sequences n the dataset are zero-padded to the maxmum length t max, the padded zero values are masked and not used n the computaton. We model the process of predctng the for each attrbuted sequence as: σw pp + b p, or o Attenton Θp = σ W pα t + b p, where σ s a sgmod actvaton functon and ˆl = Θp s the predcted. The W p and b p are both tranable n our model. 3.3 Tranng Regularzaton. We adopt multple strateges for dfferent components n our AMAS network. We emprcally select the followng regularzaton strateges n our model based on: 1. For Sequence et, we apply l 2 -regularzaton to the recurrent unt. 2. Copyrght c 2019 Copyrght for ths paper s retaned by authors

5 Table 2: Compared Methods ame Data Used Attenton ote Attrbutes o [12] Sequences o [28] S Attrbutes Sequences o [35] Sequences Yes [34] Attrbutes Sequences Yes Ths paper Attrbutes Sequences Yes Ths paper Dropout wth a rate of 0.5 s used to regularze the fully connected layer n Attrbute et. 3. Lastly, we use Dropout wth a rate of n other fully connected layers n the model. Based on our observatons, usng regularzaton on Attenton Block has no sgnfcant mpact on the performance of AMAS Optmzer. We use an optmzer that computes the adaptve learnng rates for every parameters, referred to as Adaptve Moment Estmaton Adam [15]. The core dea s to keep 1. an exponentally decayng average of gradents n the past and 2. a squared past gradents. Adam counteracts the bases as: ω t = β 1ω t β 1 m t 1 β t 1 ν t = β 2ν t β 2 m t 2 1 β t 2 where β 1 and β 2 are the decay rates, and m t s the gradent. We adopt β 1 = 0.9 and β 2 = as n [15]. Fnally, the Adam updates the parameters as: γ t+1 = γ t ρ ν t + ɛ ω t where ρ s a statc learnng rate and ɛ s a constant wth a small value to avod dvson errors, such as dvson by zero. We emprcally select ρ = Experments 4.1 Datasets. Our soluton has been motvated by use case scenaros observed at Amadeus corporaton. For ths reason, we work wth the log fles of an Amadeus [2] nternal applcaton. The log fles contan user sessons n the form of attrbuted sequences. In addton, we apply our methodology to real-world, publcly avalable Wkspeeda data [29] and Reddt data [16]. For each type of data, we sample two subsets and conduct experments ndependently. We summarze the data descrptons as follows: Amadeus data AMS-1, AMS-2 1. We sampled sx datasets from the log fles of an nternal applcaton at Amadeus IT Group. Each attrbuted sequence s composed of a user profle contanng nformaton e.g., system confguraton, offce name and a sequence of functon names nvoked by web clck actvtes e.g., logn, search ordered by tme. Wkspeeda data Wk-1, Wk-2. Wkspeeda s an onlne game requrng partcpants to clck through from a gven start page to an end page usng fewest clcks [29]. We select fnshed paths and extract several propertes of each path e.g.,, the category of the start path, tme spent per clck. We also sample sx datasets from Wkspeeda. The Wkspeeda data s avalable through the Stanford etwork Analyss Project 2 [17]. Reddt data Reddt-1, Reddt-2. Reddt s an onlne forum. Two datasets that contan the content of reddt submssons are used. The Reddt data s avalable through the Stanford etwork Analyss Project 3. We use 60% of the nstances n each dataset for the tranng and the rest 40% for testng. In the tranng, we holdout 20% of the tranng nstances for valdaton. 4.2 Compared Methods. We evaluate our two approaches, namely and and compare them wth the followng baselne methods. We summarze all compared methods used n ths research n Table 2. s bult usng a fully connected neural network to reduce the dmensonalty of the nput data, and then classfy each nstance. classfes sequences only data usng an LSTM. S utlzes the nformaton from both attrbutes and sequences. The resultng embeddngs generated by S are then used for classfcaton. bulds attenton on the sequence data for classfcaton, whle the attrbute data s not used. 4.3 Expermental Settng. Our paper focuses on mult-class classfcaton problem. We thus use accuracy as the metrc to evaluate the performance. A hgher accuracy score depcts more correct predctons of class s. For each method, we holdout 20% as the valdaton dataset randomly selected from tranng dataset. 1 Personal nformaton s not collected Copyrght c 2019 Copyrght for ths paper s retaned by authors

6 S a ASM-1, 83 classes 0.0 S S b Wk-1, 64 classes S 0.00 c Reddt-1, 140 classes d ASM-2, 83 classes e Wk-2, 64 classes f Reddt-2, 140 classes Fgure 4: Performance comparson on all sx datasets. S S For each expermental settng, we report the top-1 top-10 accuracy for each method. We ntalze our network usng the followng strateges: orthogonal matrces are used to ntalze the recurrent weghts, normalzed random dstrbuton [9] s used to ntalze weght matrces n Attrbute et, and bas vectors are ntalzed as zero vector Results. In Fgure 4, we compare the performance of our and solutons wth the other state-of-the-art methods n Table 2. acheves the best performance of top-1 accuracy on most datasets. In most cases, outperforms other solutons sgnfcantly. We also observe a sgnfcant performance mprovement by compared to other methods. That s, although the top-1 accuracy performance of s beneath that of, t stll outperforms wth sequence-only attenton and all other methods wthout attenton. The two closest compettors, the utlzng the attenton mechansm and classfyng each nstance based on only the sequental data, and S usng nformaton from both attrbutes and sequences, but wthout the help from an attenton mechansm, are outperformed by the our proposed models. 4.5 Parameter Senstvty Analyss Adaptve Samplng. As ponted out n recent work [4], adaptve samplng s capable of mprovng the effcency of the optmzaton processes by adaptng the tranng sample sze n each teraton.e., epoch. In ths set of experments, we evaluate the two models wth varyng adaptve samplng rates. We adopt the adaptve samplng functon as: τ = 1 λ τ 1 Here, τ denotes the epoch number, τ denotes the number of nstances used n the τ-th epoch and λ the rate of adaptve samplng. We choose λ = 1, 1.001, 1.005, and 1.01 n our experments, where λ = 1 means no adaptve samplng. The results presented n Fgure 5 shows that the adaptve samplng wth the above samplng rates can acheve smlar performance as the non-adaptve approach yet now wth much less tranng data Tranng wth Adaptve Samplng. Wth the contnuously ncreasng amount of tranng nstances, we expect the hstory of tranng loss to be jttery when a model encounters prevously unseen new nstances. Dfferent from prevous experments, where we use Early Stoppng strategy to avod overfttng, we now set a fxed number of 144 epochs for the model and 97 epochs for model and collect the hstory of tranng and valdaton to study the adaptve tranng strategy. In Fgure 6a, we observe the tranng wth adaptve samplng s more aggressve compared to the non-adaptve approach. From Fgure 6b we conclude that wth a hgher adaptve rate, the model more easly becomes overftted. Smlar concluson can also be made from Fgure 6d. Selectng a hgher adaptve samplng rate can shorten the tranng tme but rskng a hgher chance of overfttng. 4.6 Case Studes. Fgure 7 demonstrates the weghts of each word of ten nstances from the Reddt-2 dataset. Hgher weghts are represented wth a darker Copyrght c 2019 Copyrght for ths paper s retaned by authors

7 on-adaptve Adaptve rate=1.001 Adaptve rate=1.005 Adaptve rate=1.01 a model on AMS-1 dataset on-adaptve Adaptve rate=1.001 Adaptve rate=1.005 Adaptve rate= on-adaptve Adaptve rate=1.001 Adaptve rate= Adaptve rate=1.01 b model on Wk-1 dataset on-adaptve Adaptve rate=1.001 Adaptve rate=1.005 Adaptve rate= on-adaptve Adaptve rate=1.001 Adaptve rate=1.005 Adaptve rate=1.01 c model on Reddt-1 dataset. d model on AMS-1 dataset. e model on Wk-1 dataset. f model on Reddt-1 dataset. Fgure 5: The performance comparson between non-adaptve tranng and adaptve samplng on-adaptve Adaptve rate=1.001 Adaptve rate=1.005 Adaptve rate=1.01 Loss Epoch Loss Epoch Epoch a tran_loss of. b val_loss of. c tran_loss of. d val_loss of. Fgure 6: Comparson of the hstory of tranng and valdaton losses. Loss Loss Epoch color, whle lower weghts are represented wth a lghter color. Comparng the three cases, we fnd that the has the most polarzed weghts among the three cases. Ths may be caused by the fact that the attenton produced by s solely based on the sequences, whle and have been nfluenced by attrbute data. 5 Related Work 5.1 Deep Learnng Deep learnng has receved sgnfcant nterests n recent years. Varous deep learnng models and optmzaton technques have been proposed n a wde range of applcatons such as mage recognton [14, 32] and sequence learnng [5, 24, 33, 22]. Many of these applcatons nvolve the learnng of a sngle data type [5, 24, 33, 22], whle some applcatons nvolve more than one data type [14, 32]. The applcaton of deep learnng n sequence learnng s popular wth one of the most popular works, sequence-to-sequence [24], usng a long short-term memory model n machne translaton. The hdden representatons of sentences n the source language are transferred to a decoder to reconstruct n the target language. The dea s that the hdden representaton can be used as a compact representaton to transfer smlartes between two sequences. Mult-task learnng [18] examnes three mult-task learnng settngs for sequence-to-sequence models that am to share ether an encoder or a decoder n an encoder-decoder model settng. Although the above work s capable of learnng the dependences wthn a sequence, none of them focuses on learnng the dependences between attrbutes and sequences. Multmodal deep neural networks [14, 23, 32] are desgned for nformaton sharng across multple neural networks, but none of these works focuses on the attrbuted sequence classfcaton problem we target here. 5.2 Attenton etwork. Attenton network [20] has ganed a lot of research nterest recently. The attenton network has been appled n varous tasks, ncludng mage captonng [32, 21], mage generaton [11], speech recognton [6] and document classfcaton [34]. The goal of usng an attenton network n these tasks s to make the neural network focus on the nterestng Copyrght c 2019 Copyrght for ths paper s retaned by authors

8 How reddt seems to reacts whenever I share a mlestone that I passed How I felt when forgot to put spoler n a comment What r/askreddt dd to me when I asked somethng about 9gag The reacton face when express a vew for relgon When I see a rage comc out of ts sub Reddt Sttng as a /new knght of /r/gamng seeng all the nsanely twsted shadow planet flash sale posts How I act when I see a How I feel when post n /r/funny Pretty much what happened to my pxel art post on /r/mnecraft Seeng the same thng posted multple tmes by the same person n new Seeng somethng you posted yesterday on the front page posted by someone else a How reddt seems to reacts whenever I share a mlestone that I passed How I felt when forgot to put spoler n a comment What r/askreddt dd to me when I asked somethng about 9gag The reacton face when express a vew for relgon When I see a rage comc out of ts sub Reddt Sttng as a /new knght of /r/gamng seeng all the nsanely twsted shadow planet flash sale posts How I act when I see a How I feel when post n /r/funny Pretty much what happened to my pxel art post on /r/mnecraft Seeng the same thng posted multple tmes by the same person n new Seeng somethng you posted yesterday on the front page posted by someone else b How reddt seems to reacts whenever I share a mlestone that I passed How I felt when forgot to put spoler n a comment What r/askreddt dd to me when I asked somethng about 9gag The reacton face when express a vew for relgon When I see a rage comc out of ts sub Reddt Sttng as a /new knght of /r/gamng seeng all the nsanely twsted shadow planet flash sale posts How I act when I see a How I feel when post n /r/funny Pretty much what happened to my pxel art post on /r/mnecraft Seeng the same thng posted multple tmes by the same person n new Seeng somethng you posted yesterday on the front page posted by someone else c Fgure 7: Weghts of each words from 10 nstances n Reddt-2. Hgher weghts are darker. parts of each nput, such as, a small regon of an mage or words that are most helpful to classfyng documents. There are varatons of attenton networks, ncludng herarchcal attenton [34] and dual attenton [21]. 5.3 Sequence Mnng. Recent work n sequence mnng area ams to fnd the most frequent subsequence patterns [19, 8]. Several recent works [3, 19] focus on fndng the most frequent subsequence that meets certan constrants. That s, they fnd the set of sequental patterns satsfyng varous lngustc constrants e.g., syntactc, symbolc. Many sequence mnng works focus on frequent sequence pattern mnng. Recent work n [19] targets fndng subsequences of possble non-consecutve actons constraned by a gap wthn sequences. [7] ams at solvng pattern-based sequence classfcaton problems usng a parameter-free algorthm from the model space. [8] bulds a subsequence nterleavng model for mnng the most relevant sequental patterns. However, none of the above works supports attrbute data alongsde the sequences, nor do they classfy attrbuted sequences. 6 Concluson In ths paper, we propose a AMAS framework wth two models for classfyng attrbuted sequences. Our and models progressvely ntegrate the nformaton from both attrbutes and sequences whle weghtng each tem n the sequence to mprove the classfcaton accuracy. Expermental results demonstrate that our models sgnfcantly outperform state-of-the-art methods. 7 Acknowledgement We would lke to thank the Amadeus IT Group for the generous supports n fundng, computng resources and data access. References [1] Z. Akata, F. Perronnn, Z. Harchaou, and C. Schmd, Label-embeddng for attrbute-based classfcaton, n Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton CVPR, 2013, pp [2] Amadeus, Amadeus IT Group. com. Accessed: Copyrght c 2019 Copyrght for ths paper s retaned by authors