Iterative Reorganization with Weak Spatial Constraints: Solving Arbitrary Jigsaw Puzzles for Unsupervised Representation Learning

Transcription

1 Itrativ Rorganization with Wak Spatial Constraints: Solving Arbitrary Jigsaw Puzzls for Unsuprvisd Rprsntation Larning Chn Wi, Lingxi Xi,, Xutong Rn, Yingda Xia, Chi Su, Jiaying Liu, Qi Tian, Alan L. Yuill Pking Univrsity Johns Hopkins Univrsity Noah s Ark Lab, Huawi Inc. Kingsoft Cloud wichn8@pku.du.cn 98808xc@gmail.com tonghln@pku.du.cn yxia@jhu.du suchi@kingsoft.com liujiaying@pku.du.cn tian.qi@huawi.com alan.l.yuill@gmail.com Abstract Larning visual faturs from unlabld imag data is an important yt challnging task, which is oftn achivd by training a modl on som annotation-fr information. W considr spatial contxts, for which w solv so-calld jigsaw puzzls, i.., ach imag is cut into grids and thn disordrd, and th goal is to rcovr th corrct configuration. Existing approachs formulatd it as a classification task by dfining a fixd mapping from a small subst of configurations to a class st, but ths approachs ignor th undrlying rlationship btwn diffrnt configurations and also limit thir applications to mor complx scnarios. This papr prsnts a novl approach which applis to jigsaw puzzls with an arbitrary grid siz and dimnsionality. W provid a fundamntal and gnralizd principl, that wakr cus ar asir to b larnd in an unsuprvisd mannr and also transfr bttr. In th contxt of puzzl rcognition, w us an itrativ mannr which, instad of solving th puzzl all at onc, adjusts th ordr of th patchs in ach stp until convrgnc. In ach stp, w combin both unary and binary faturs of ach patch into a cost function judging th corrctnss of th currnt configuration. Our approach, by taking similarity btwn puzzls into considration, njoys a mor fficint way of larning visual knowldg. W vrify th ffctivnss of our approach from two aspcts. First, it solvs arbitrarily complx puzzls, including high-dimnsional puzzls, that prior mthods ar difficult to handl. Scond, it srvs as a rliabl way of ntwork initialization, which lads to bttr transfr prformanc in visual rcognition tasks including classification, dtction and sgmntation.. Introduction Dp larning spcially convolutional nural ntworks has bn boosting th prformanc of a wid rang of This work was partly don whn th first author was intrning at Johns Hopkins Univrsity and Huawi Noah Ark s Lab. Lingxi Xi is th corrsponding author. Figur. W study th problm of solving jigsaw puzzls for visual rcognition. Compard to th prvious work [] working on,000 fixd configurations of puzzls, our mthod can gnraliz to arbitrary configurations of D and D puzzls. applications in computr vision []. Ths statisticsbasd approachs build hirarchical structurs which contain a larg numbr of nurons, so that visual knowldg is larnd by fitting labld training data []. Howvr, annotating a larg-scal datast is oftn difficult and xpnsiv. Thrfor, wakly suprvisd or unsuprvisd larning has attractd a lot of rsarch attntions [8, ]. Ths approachs ar oftn built on som naturally xisting constraints such as tmporal consistncy [7], spatial rlationship [8] and sum-up quations []. Such information, though bing wak, constructs loss functions without rquiring annotations, and ntworks pr-traind in this way can ithr b usd for wak visual fatur xtraction [7] or fin-tund in a standalon suprvisd larning procss towards bttr rcognition prformanc [0]. In this work, w focus on a spcific way of xploiting spatial rlationship, which is to solv so-calld jigsaw puzzls in unlabld imag data [,,, ]. Ths approachs work by cutting an imag into a grid, say,, of patchs and thn disordring thm as training data, with

2 th goal to rcovr its corrct spatial configuration. Exampls ar shown in Figur. Thus, in ordr to achiv this goal, th ntwork should hav th ability to captur som smantic information,.g., larning th concpt of car and ground, though not labld, knowing that car always appars abov ground. Tchnically, ths approachs simply assignd ach configuration a uniqu ID, so that puzzl rcognition turns into a plain classification problm. W point out two major drawbacks of this stratgy. First, a plain classifir taks an assumption that all configurations ar qually similar to ach othr, which is not tru vn whn intr-class distancs ar maximizd using a grdy algorithm []. This brings a ngativ impact to rprsntation larning. Scond, th numbr of paramtrs rquird for plain classification incrass linarly with th numbr of configurations, so that it is difficult to dal with all possibl configurations du to th risk of ovr-fitting. For xampl, thr ar 9! =,880 possibl configurations for a puzzl, but th original approach [] rachd th bst prformanc at,000 and obsrvd ovr-fitting whn this numbr continus growing. Both of ths drawbacks limit us from gnralizing this approach to mor complx puzzls. On th othr hand, solving complx puzzls is spcially usful for som aras such as mdical imaging analysis, in which it is difficult to pr-train D ntworks [, 8] in D scnarios, yt a rasonabl initialization hlps a lot on training stability and tsting prformanc. An mpirical study of this topic can b found in Sction.. In this papr, w xtnd th ability of such approachs by allowing to solv arbitrary jigsaw puzzls, i.., th puzzls ar not constraind by a pr-dfind st of configurations. Our improvmnt lis in two parts. First, w allow th jigsaw puzzls to hav an arbitrary configuration rathr than bing limitd in a fixd st. Scond, w introduc an itrativ solvr built upon wak spatial constraints to rplac th dirct classifir usd prviously. As a rsult, our approach xtnds th ability of rprsntation larning basd on jigsaw puzzls and transfrs wll to D data. To this nd, w formulat puzzl rcognition into an optimization problm which involvs a st of unary and binary trms, with ach unary trm indicating whthr a spcifid patch is locatd at a spcifid position, and ach binary trm masuring whthr two patchs should hav a spcifid rlativ position. Ths trms ar dtrmind by a dp ntwork backbon so that th ntir systm can b traind in an nd-to-nd mannr. In both training and tsting, w allow th first trial not to find th corrct configuration, in which cas w itrativly adjust th configuration according to prdiction until convrgnc. W valuat our approach in both puzzl rcognition and transfr larning. Th puzzl solvr is traind on ILSVRC0 training st [] and tstd on th validation st, both of which do not contain class labls. Our approach solvs arbitrary jigsaw puzzls with rasonabl accuracy, whil th prior approachs can only work on a limitd st of puzzls. Thn, w transfr th pr-traind modl to b fin-tund on th Pascal VOC 007 datast [] for imag classification and objct dtction. Eithr larning from mor complx puzzls or achiving highr accuracy in puzzl rcognition boosts transfr larning prformanc, which vrifis our motivation. Finally, w apply our approach to initializ a D ntwork with unlabld mdical data, and vrify its ffctivnss in sgmnting an abdominal organ from CT scans. Th rmaindr of this papr is organizd as follows. Sction brifly rviws rlatd work, and Sction dscribs th proposd approach. Aftr xprimnts shown in Sction, w draw our conclusions in Sction.. Rlatd Work Dp nural ntworks hav bn playing an important rol in modrn computr vision systms. With th availability of larg-scal datasts [7] and powrful computational dvic such as GPUs, rsarchrs hav dsignd ntwork structurs with tns [,, ] or hundrds [, 7] of layrs towards bttr rcognition prformanc. Also, th ntworks pr-traind on ImagNt wr transfrrd to othr rcognition tasks by ithr xtracting visual faturs dirctly [9,, 7] or bing fin-tund on a nw loss function [7, 8]. Dspit thir ffctivnss, ths ntworks still strongly rly on labld imag data. Howvr in som aras such as mdical imaging, data collction and annotation can b xpnsiv, tim-consuming, or rquiring xprtis. Thus, thr has bn fforts to dsign unsuprvisd [8, ] or wakly suprvisd [8] approachs which larn visual knowldg from unlabld data, or smisuprvisd larning algorithms [, ] which aim at combining a limitd amount of labld data and a larg corpus of unlabld data towards bttr prformanc. It has bn vrifid that unsuprvisd pr-training hlps suprvisd larning spcially dp larning [0]. Th ky factor to larning from unlabld data is to stablish som kind of prior, or som wak constraints that naturally xist, i.., no annotations ar rquird. Such prior can b ithr () mbddd into th ntwork architctur or () ncodd as a wak suprvision to optimiz th ntwork. For th first typ, rsarchrs dsignd clustring-basd approachs to optimiz visual rprsntation [0, ], as wll as gnrator-basd approachs [, ]. This papr mainly considrs th scond typ which, in comparison to th first typ, is much asir in algorithmic dsign. Typical xampls includ tmporal consistncy which assums that nighboring vido frams contain similar visual contnts [7, ], th tmporal ordr in th contxt of vido [9,, ], spatial rlationship btwn som pairs of unlabld patchs [8], larning an additiv function

3 on diffrnt rgions as wll as th ntir imag [], tc. Among ths priors, spatial contxts ar widly blivd to contain rich information which a vision systm should b abl to captur. Going on stp byond modling patch rlationship [8], rsarchrs dsignd so-calld jigsaw puzzls [,,, ] which ar mor complx so that th ntworks ar bttr traind by larning to solv thm. Rsarchrs blivd that larning from ths waklysuprvisd cus can hlp visual rcognition, bcaus many problms ar indd built on undrstanding and intgrating this typ of information. Rgarding spatial contxts, a wid rang of rcognition tasks can bnfit from undrstanding th rlativ position of two or mor patchs, such as imag classification [], smantic sgmntation [] and parsing [], tc.. Our Approach.. Problm and Baslin Solution Th problm of puzzl rcognition assums that an imag is partitiond into a grid (.g., ) of patchs and thn disordrd. Th task is to rcovr th original configuration (i.., patchs ar ordrd in th natural form). To accomplish this task, th ntwork nds to undrstand what a patch contains as wll as how two or mor patchs ar rlatd to ach othr (.g., in a car imag, a whl is oftn locatd to th top of th ground). Thrfor, w xpct this task to tach a ntwork both intra-patch and intr-patch information, which w formulat as unary trms and binary trms, rspctivly. W first dfin th trminologis usd in this papr. Lt I b an imag, which is partitiond into W H patchs. Each patch, dnotd as i x,y (0 x < W, 0 y < H), is assignd a uniqu ID a x,y {0,,..., W H } according to its original position,.g., th row-major policy givs a x,y = x + yw. Aftr that, all patchs ar randomly disordrd, and w us c x,y to dnot th ID ownd by th patch that currntly occupis th (x, y) position. All c x,y valus ) W,H x=0,y=0. compos a configuration, dnotd as c = ( c x,y Thr ar in total (W H)! diffrnt configurations, composing th configuration st C that C = (W H)!. Our goal is to prdict th corrct configuration c C. For this purpos, a ntwork structur with two parts was constructd []. Th ntwork backbon M B : f x,y = ( f i x,y ; θ B) is built upon ach individual patch, and outputs a st of faturs for th ntwork had M H : c = g (F; θ H) to produc th final output c = (c x,y ) W,H x=0,y=0, whr F = (f x,y ) W,H x=0,y=0 is th ordrd concatnation of patch ( faturs. In practic, f ; θ B) is oftn borrowd from xisting ntwork architcturs [,, ], whil g ( ; θ H) is oftn mor intrsting to invstigat. In th prior work [, ], th ntwork had workd by constraining th numbr of possibl configurations, say K =,000 out of 9!, which ar randomly sampld from C using a grdy algorithm to guarant th Hamming distanc btwn ( any two configurations is sufficintly larg. Thn, f ; θ H) was dsignd to b a K-way classifir, implmntd as a fully-connctd layr. Th purpos of this dsign was mainly to control th numbr of paramtrs of th classifir (proportional to K) so as to prvnt ovrfitting, but w argu that it largly limits th modl from bing applid mor complx scnarios lik D puzzls, whil it was blivd that larning from a hardr task can lad to a strongr ability []. This motivats us to propos a nw approach in which th numbr of configurations can b arbitrarily larg whil th numbr of paramtrs rmains unchangd. W will s latr that th ssnc bhind this motivation is to us wak cus with an itrativ algorithm towards a mor compact rprsntation and a safr larning procss... Solving Jigsaw Puzzls with Wak Cus W dsign a ntwork had to larn wak spatial constraints. By wak w ar comparing this stratgy with th aformntiond K-way classifir that prdicts th configuration of th ntir puzzl all at onc. Instad, w considr an indirct cost function S(I, c) which outputs a cost that patch i x,y or quivalntly fatur f x,y is locatd at position c x,y, and thus th most probabl configuration is dtrmind by arg max c {S(I, c)}. S(I, c) is composd of two parts, namly, unary trms and binary trms. Each unary trm provids cus for th absolut position of a patch, and ach binary trm provids cus for th rlativ position of two patchs. Mathmatically, S(I, c) S(F, c) = (x,y)p (f x,y, c x,y F) + (x,y ) (x,y ) p (f x,y, f x,y, c x,y, c x,y ). () Hr, p (f x,y, c x,y F) is a unary trm which masurs how likly that patch f x,y is locatd at position c x,y, and p (f x,y, f x,y, c x,y, c x,y ) is a binary trm masurs how likly that patchs f x,y and f x,y hav th spatial rlationship indicatd by c x,y and c x,y. Each unary trm is computd basd on F, th ovrall variabl containing fatur vctors of all patchs, bcaus th position of ach [] obsrvd that stting a largr K lads to prformanc drop in transfr xprimnts, and xplaind it as th ntwork gts confusd by vry similar jigsaw puzzls. Howvr, as shown in xprimnts (s Sction.), our approach works wll in th ntir puzzl st C, i.., K = 9! =,880, which implis that th prformanc drop may du th larg numbr of paramtrs.

4 σ Input Imag I sharing wights concatnation softmax Input Configuration c 0 8 Backbon Backbon Backbon Backbon Fatur f x,y Fatur f x,y Fatur f x,y Fatur f x,y Each Two Faturs Concatnation All Faturs Concatnation FC Layrs FC Layrs Prdictd Configuration Fatur c Rorganization σ σ σ σ σ Rlativ Position Vctor V 9 Configuration Matrix U 9 9 Max-Cost Max-Matching Binary Trm p Cost Function S I, c Unary Trm p Figur. Th ovrall structur (bst viwd in color). Each training imag (without smantic annotations) is randomly croppd, disordrd and fd into puzzl rcognition ntwork. Two typs of loss trms (unary and binary) ar computd and summd into th final cost function S(I, c). Th training procss continus until th puzzl is compltly corrct or a maximal numbr of rounds is achivd. patch f x,y dpnds on th visual mssags dlivrd by othr patchs. Th binary trms, on th othr hand, do not hav such a dpndncy. In practic, th unary trms ar formulatd in a matrix U with W H W H lmnts, ach of which, U a,c, indicats th cost obtaind by putting th spcifid patch with ID a at a spcifid position with ID c. This is implmntd by a fully-connctd layr btwn F and ths (W H) lmnts, paramtrizd by θ U. W prform th softmax function ovr all lmnts in ach row, so that th scors corrsponding to ach patch sum to. Thn, ach unary trm is th log-liklihood of th scor at a spcifid position: ( p (f x,y, c x,y, F) = ln U F; θ U). () a x,y,c x,y For ach binary trm involving f x,y and f x,y, w build anothr mapping from ths two vctors to a 9- dimnsional vctor, with ach indx indicating th probability that th spatial rlationship of f x,y and f x,y blongs to on of th 9 possibilitis, namly, th first patch is locatd to th top, bottom, lft, right, top-lft, top-right, bottomlft, bottom-right of th scond patch or non of th abov happns. Similarly, this is implmntd using anothr fullyconnctd layr btwn f x,y f x,y ( dnots concatnation) and a 9-dimnsional vctor paramtrizd by θ V Idally, th lmnts in ach column should also sum to, but it is mathmatically intractabl if w hop to kp th ratio btwn all lmnts. Thr ar two argumnts. First, aftr normalizing scors in ach row, w find that thr oftn xists on major lmnts in ach column, and th sum of ach column is clos to. Scond, w add an additional l loss trm btwn th sum of ach column and, but only obsrv to minor changs in ithr puzzl rcognition accuracy or transfr larning prformanc. followd by a softmax activation ovr ths 9 numbrs. W. dnot r x,y,x,y = r(cx,y, c x,y ) {0,,..., 8} as th rlativ position typ btwn f x,y and f x,y, so that w can writ th binary trm as: p (f x,y, f x,y, c x,y, c x,y ) = ( ln V f x,y, f x,y ; θ V). () r x,y,x,y Compard to a plain classifir assigning a class indx to ach puzzl, th amount of paramtrs rquird by our approach is rducd. Tak a puzzl as an xampl, and w assum that F contains D lmnts. On th on hand, th K-way classifir rquirs KD paramtrs (a typical stting [] is K =,000) which grows linarly with K. On th othr hand, our approach rquirs (W H) D paramtrs for th unary trms, and 9D paramtrs for th binary trms. Th total numbr of paramtrs, ( W H + 9 ) D (.g., 90D for a puzzl), is largly rducd and dos not incras with K. Consquntly, our approach is asir to b applid to th scnario with a largr st of (.g., all 9! possibl) configurations. This advantag is vrifid in xprimnts. Last but not last, thr ar many othr ways of using wak spatial constraints to formulat S(I, c) w just provid a practical xampl... Optimization: Itrativ Rorganization W aim at optimizing S(F, c) with rspct to ntwork paramtrs θ U, θ V and configuration c. Howvr, not that c is a discrt variabl which cannot b optimizd by gradint dscnt. So w apply diffrnt stratgis in training and tsting.

5 In th training stag, w know th ground-truth configuration c, so th optimization bcoms: arg min θ U,θ V S(F, c ). () This is implmntd by stting th suprvision signal accordingly, i.., th corrct clls ar filld up with whil othrs with 0, and using stochastic gradint dscnt. Not that ach unary trm dpnds on th ordr of input patchs. To sampl mor training data as wll as adjust data distribution (xplaind latr), w introduc itration to th training stag. Dnot th input configuration as c (0) = c, and th corrsponding fatur as F (0). In ach itration, with fixd θ U and θ V, w maximiz S(F, c) with rspct to c: c = arg min S c ( F, c (0)), () and us c to find th nxt input c (), so that applying c to c () obtains c (0),.g., if c is prfct, thn c () corrsponds to th original configuration that vry patch is placd at th corrct position. This procss continus until convrgnc or a maximal numbr of itrations is rachd. Th losss with rspct to θ U and θ V ar accumulatd, avragd, and back-propagatd to updat ths two paramtrs. Th sam stratgy, itration, is usd at th tsting stag to solv jigsaw puzzls, with th only diffrnc that no gradint backpropagation is rquird. It rmains a problm to solv Eqn (). This is a combinatoric optimization problm, as c can only tak (W H)! discrt valus which indicat th ntris in U and V that ar summd up. Thr is obviously no closd form solutions to maximiz S(F, c), yt numrating all (W H)! possibilitis is computationally intractabl spcially whn th puzzl siz bcoms larg. A possibl solution lis in approximation, which first switchs off all binary trms, so that th optimization bcoms choosing W H ntris from a W H W H matrix with a maximal sum, but no two ntris can appar in th sam row or column (this is a maxcost-max-matching problm, and th bst solution c can b found using th Hungarian algorithm); thn numrats all possibilitis within a limitd Hamming distanc from c and chooss th on with th bst ovrall cost S(F, c). Finally, w discuss stratgy of introducing itration to solv this problm. Mathmatically, Eqn () is a fixdpoint modl [], i.., th output variabl c also impacts F and thus S(F, c), so itration is considrd a rgular way of optimizing it. Howvr, th rols playd by itration ar diffrnt in training and tsting. In th training stag, aftr ach itration, w shall xpct th configuration to b adjustd closr to th ground-truth. Thrfor, if w tak th input configuration fd into ach round as an individual cas, thn th distribution of input data is changd by itration, and th cass that ar mor similar to th groundtruth ar mor likly to b sampld. Thrfor, in th tsting stag, w can xpct th itration to improv puzzl rcognition accuracy, bcaus as th itration continus, th input puzzl gts closr to th ground-truth by statistics, and our modl ss mor training data in this scnario and is strongr. W show a typical xampl in Figur, in which w can obsrv how itration gradually prdicts th corrct configuration.. Exprimnts.. Jigsaw Puzzl Rcognition W follow [] to train and valuat puzzl rcognition on th ILSVRC0 datast [], a subst of th ImagNt databas [7]. W train th modl using all th.m training imags and tst it on th validation st with 0K imags, both of which do not contain class annotations. In th training stag, w pr-procss th imags to prvnt th modl from bing disturbd by pixl-lvl information. W first dtrmin th siz of puzzls,.g., W H, and thn rsiz ach input imag into 8W 8H and partition it vnly into a W H grid. In ach 8 8 imag, w randomly crop a subimag as th patch fd into th puzzl rcognition ntwork. To maximally rduc th possibility that low-lvl information is usd, w furthr horizontally flip ach input patch with a probability of 0% and subtract man valu from ach channl w do not prform othr data augmntation tchniqus bcaus thy ar lss likly to appar in ral data. In practic, flip augmntation brings consistnt accuracy gain to transfr larning tasks though w obsrv significant accuracy drop in puzzl rcognition (s Tabl ). Th backbon of our puzzl ntwork is borrowd from two popular architcturs, namly, an 8-layr AlxNt [] and two dp RsNts [] with 8 and 0 layrs. W do not valuat VGGNt [] as in [, ] bcaus it is mor difficult to initializ and producs lowr accuracy than RsNts. Th outputs of th first layr with a spatial rsolution of (i.., fc in AlxNt and avg-pool in RsNts) ar fd into a,0-way fully-connctd layr and th output is takn as f x,y, followd by our dsignd layrs for xtracting unary and binary trms for puzzl rcognition. All ths ntworks ar traind from scratch. W us th SGD optimizr and a total of 0K itrations (minibatchs) for AlxNt and 0K for RsNts. Each batch contains puzzls. On four NVIDIA Titan-V00 GPUs, th training tims on AlxNt, RsNt8 and RsNt0 ar 0, 0 and 0 hours, rspctivly. In th tsting stag, to rduc randomization factors, w switch off randomization in patch cropping and data augmntation, with ach patch croppd at th cntr of th 8 8 filds and not flippd. Rsults ar summarizd in Tabl. W first valuat puzzl rcognition accuracy. For ach imag, thr ar 9! =,880 possibl puzzls, so

6 Stting Pr-training Options Puzzl Rcognition Pascal VOC 007 ID Siz Backbon Labl Unary Binary Mirror Corrct D Classifi. Dtc. (a) AlxNt (b) AlxNt.. (c) AlxNt (d) AlxNt () AlxNt (f) AlxNt (g) RsNt (h) RsNt8..8 (i) RsNt (j) RsNt (k) RsNt (l) RsNt (m) RsNt0.8. (n) RsNt (o) RsNt (p) RsNt Comptitors with Diffrnt Backbons, Pr-training Cus and Sttings Rf. Yar Backbon Dscription of unsuprvisd training Classifi. Dtc. [8] 0 AlxNt Dtrmining th rlativ spatial position of two patchs.. [7] 0 AlxNt Unsuprvisd tracking in vidos. 7. [] 0 AlxNt jigsaw puzzls with a,000-way plain classifir 7.7. [] 07 RsNt Prdicting color from gray-scal intnsity 77. [] 07 AlxNt Counting visual primitivs in subrgions 7.7. [] 08 AlxNt Classifying aftr clustring itrativly 7.7. [] 08 AlxNt Prdicting D imag rotations 7.0. [0] 08 AlxNt [8] with nhancmnt tchniqus 9..8 [] 08 VGGNt [] with knowldg distillation and noisy patchs 7.. [9] 08 AlxNt Prdicting surfac normal, dpth, and instanc contour 8.0. Tabl. Puzzl rcognition and transfr larning accuracy (%). In th pr-training options, labld mans to us th annotatd ILSVRC0 training st to pr-train a ntwork. Th instancs without any imply that Pascal VOC 007 tasks ar traind from scratch. W also compar with prior approachs, som of which hav diffrnt knowldg sourcs, ntwork backbons and training stratgis. W rport th most powrful ntwork backbon usd in ach papr. Th works with puzzl rcognition ar highlightd in grn. random guss givs a 0.000% accuracy. With only unary trms (Eqn can b solvd by th Hungarian algorithm), all ntwork backbons achiv ovr 0% accuracy without mirror augmntation, which shows that wak visual cus can b combind to infr global patch contxts. On top of this baslin, w invstigat th impact of othr four options. First, adding binary trms consistntly improvs puzzl rcognition accuracy, arguably du to th additional contxtual information, which is spcially usful in dtrmining th rlativ position of two nighboring patchs. Scond, mirror augmntation rducs puzzl rcognition accuracy dramatically in both training and tsting, but as w will s latr, this stratgy improvs th gnralization ability of our pr-traind modls to othr rcognition tasks. Third, compard with puzzls, jigsaw puzzls ar naturally mor difficult to solv, but thy also forc th modl to larn mor visual knowldg and thus hlp transfr larning, as shown in our latr discussions. Fourth, th abov phnomna rmain th sam as th ntwork backbon bcoms strongr, on which both puzzl rcognition and transfr visual rcognition bcoms mor accurat. As a sid commnt, w point out that convntional puzzl rcognition approachs with plain classification [, ] oftn achivd highr puzzl rcognition accuracy in a limitd class st. With modls traind with our approach (Lin () in Tabl ) w numrat th,000 classs gnratd with algorithm providd by [] and find th maximal S(F, c), so as to mimic th bhavior of plain classification. Our modls with AlxNt rports a 0.% puzzl rcognition accuracy which is lowr than 7% rportd in []. Howvr, our approach njoys bttr transfr ability, as w will s in latr xprimnts. In addition, th prformanc of [] dgnrats with incrasd puzzl siz, as th fraction of xplord puzzls bcoms smallr, yt th waknss of ignoring undrlying rlationship btwn diffrnt configurations bcoms mor significant and harmful. From this prspctiv, th advantag of solving arbitrary puzzls

7 Input Imag Input Configuration Round Round Round Round Hamming Dist Corrct 0.0%.%.%.% 7.% Figur. Two xampls of diffrnt difficultis in itrativ puzzl rcognition (bst viwd in color). Each digit to th lowr-lft cornr of ach patch is th corrsponding patch ID. For ach round, w also rport puzzl rcognition statistics ovr th ntir tsting st. Unary Binary Mirror conv conv conv conv conv Noroozi t al. [] Noroozi t al. [] Tabl. ILSVRC0 (top) and Placs0 (bottom): classification accuracy with linar classifirs on top of frozn convolutional layrs of AlxNt. bcoms clarr. Th sam phnomnon also happns in D puzzls (Sction.). Som statistics for our modl with RsNt0 (Lin (p) in Tabl ) as wll as two typical xampls ar shown in Figur (on is difficult and not solvd). W can obsrv how th disordrd patchs ar rorganizd with wak spatial cus throughout an itrativ procss. As an ablation study, w xprimnt with fwr numbrs of maximal itrations, namly, and 0 instad of 0, but achiv lowr accuracis in both puzzl rcognition and transfr larning tasks. This justifis our hypothsis that itration, togthr with wak spatial cus, provids a mild way of unsuprvisd larning, which bttr fits stat-of-th-art dp ntworks... Transfr Larning Prformanc Nxt, w invstigat how wll our modls pr-traind on puzzl rcognition transfr to othr visual rcognition tasks. Following th convntions [, ], w valuat classification and dtction tasks on th Pascal VOC 007 datast []. All pr-traind ntworks undrgo a standard fin-tuning flowchart, with a plain classifir and Fast- RCNN [] bing usd as ntwork hads, rspctivly. W do not lock any layrs in our ntwork, bcaus this oftn lads to wors transfr prformanc as shown in prior approachs [,, ]. Rsults ar summarizd in Tabl. W can obsrv som intrsting phnomna. First, transfr rcognition prformanc gos up with th powr of ntwork backbons, which shows th ability of our approach to tap th potntial of dp ntworks. Scond, both unary and binary trms contribut to transfr accuracy and thy ar complmntary. Third, mirror augmntation harms puzzl rcognition but improvs transfr larning, bcaus it allviats th chanc that dp ntworks borrow low-lvl pixl continuity in solving th jigsaw puzzls which falls into th catgory of ovr-fitting and hlps transfr rcognition vry littl. Hr is a sid not. It was suggstd in [] that forcing th ntwork to discriminat vry similar puzzls (.g., only a pair of patchs ar rvrsd) oftn lads to accuracy drop bcaus th modl can focus too much on local pattrns. In th contxt of using AlxNt to solv puzzls, w study diffrnt numbrs of configurations, i.., % (,9), 0% (,88) and all (9! =,880) possibl puzzls. W find that our approach rports th bst transfr accuracy at th last option, whil using smallr numbrs of configurations lads to slightly wors prformanc. Hnc, w mak th following conjctur: it is indd th largr numbr of paramtrs in a plain classifir, rathr than solving vry similar puzzls, that causs transfr prformanc drop. In th last part, w valuat th quality of faturs xtractd from th pr-traind modls dirctly. W implmnt th standard xprimnts on th ILSVRC0 datast [] and Placs0 [] with a linar classifir on top of frozn 7

8 convolutional layrs of AlxNt. Rsults ar summarizd in Tabl. Our approach, particularly with binary trms and mirror augmntation, shows highr numbrs with conv faturs. This is in lin with our motivation, namly, solving arbitrary puzzls with wak spatial constraints indd improvs mid-lvl rprsntation larning... Gnralization to D Ntworks Finally, w apply our modl to a D visual rcognition task. W study mdical imaging analysis, an important prrquisit for computr-assistd diagnosis (CAD). Most mdical data ar volumtric, and rsarchrs hav proposd som D ntwork architcturs [, 8]. Compard to D ntworks [0, ], D ntworks njoy th bnfit of sing mor contxtual information, but still suffr th drawback of missing a pr-traind modl. Du to th common situation that th amount of training data is limitd, ths D ntworks oftn hav a rlativly unstabl training procss and somtims this downgrads thir tsting accuracy [9]. Our approach provids a solution for initializing D ntworks with jigsaw puzzls. W invstigat th NIH pancras sgmntation datast [], which contains 8 cass. W partition it into folds (around 0 cass in ach fold), us thr of thm to train a sgmntation modl and tst it on th rmaining on. To construct jigsaw puzzls, w ithr dirctly us th training sampls in th NIH datast, or rfr to anothr public datast namd Mdical Sgmntation Dcathlon (MSD) th pancras tumour subst with 8 training cass. For all th data usd for jigsaw puzzls, w do not us any pixl-lvl annotations though thy ar providd. W randomly crop volums within ach cas, and cut it vnly into two puzzl sizs, namly, pics with a subvolum croppd within ach cll, or pics with a subvolum croppd within ach cll. A typical xampl is shown in Figur. W randomly disordr ths patchs using all 8! or 7! possibl configurations, and th task is to rcovr th original configuration. W us VNt [8] as th baslin (only th down-sampling layrs ar usd in this stag), and comput th unary trms in an 8 8 or 7 7 matrix. W switch off th binary trms basd on th considration that on patch has nighbors in th D spac which maks prdiction ovr-complicatd. Now w rcovr th complt VNt structur with randomly-initializd up-sampling layrs and start training on th NIH training st ( cass) and its substs. Rsults ar shown in Tabl rvaling som usful knowldg. First, pr-training on jigsaw puzzls hlps sgmntation spcially in th scnarios of fwr training data. Scond, visual knowldg larnd in this mannr can transfr across diffrnt datasts rgardlss of th diffrnt distributions in intnsity causd by th scanning dvic. Third, th Data Scratch Pr-traind on NIH Pr-traind on MSD 0% % % Tabl. Pancras sgmntation accuracy (DSC, %) with diffrnt amounts of training data and diffrnt initialization tchniqus. In ach group, th accuracy is avragd ovr 0 tsting cass. modl pr-traind on NIH with puzzls prforms wors than th from-scratch on whn tstd on 00% data. W conjctur that solving such puzzls provids a bttr initialization, but th domain gap btwn solving puzzls and smantic sgmntation cancls out th positiv ffct of initialization. This raiss th ncssity of solving mor complx puzzls which agrs with our motivation. Fourth, constructing largr and thus mor difficult puzzls improvs th basic ability of ntworks. This shows th valu of our rsarch it is unlikly for th baslin approach to sufficintly xplor th spac of puzzls, which has 7!. 0 8 diffrnt configurations. Sampling,000 configurations with th grdy algorithm dscribd in [] downgrads sgmntation accuracy to 7.% and 80.8% with 0% and 00% data, rspctivly.. Conclusions This work gnralizs th framwork of jigsaw puzzl rcognition which was prviously studid in a constraind cas. To this nd, w chang th ntwork had from a plain K-way classifir to a combinatoric optimization problm which uss both unary and binary wak spatial cus. This stratgy rducs th numbr of larnabl paramtrs in th modl, and thus allviats th risk of ovr-fitting. Th incrasd flxibility of pr-training allows us to apply our approach to a wid rang of transfr larning tasks, including dirctly using it for fatur xtraction, and gnralizing it to th D scnarios to provid an initialization for othr tasks,.g., mdical imaging sgmntation. Our study rvals th as and bnfits of larning to rcogniz wak visual cus in unsuprvisd larning, in which th ky problm oftn lis in finding a compact way of rprsnting knowldg,.g., dcomposing th ntir puzzl into unary and binary trms. W point out that th xploration of unsuprvisd larning is still far from th nd. In th futur, w will also apply our mthod to lss structurd data such as graphs [0] and mor structurd data such as vidos [9], and xplor its ability of larning visual knowldg in an unsuprvisd mannr. Acknowldgmnts This papr was supportd by ONR N W thank Wichao Qiu, Chnxi Liu, Zhuotun Zhu, Siyuan Qiao, Yutong Bai and Angtian Wang for instructiv discussions. 8

9 Rfrncs [] J. Aghajanian, J. Warrll, S. J. Princ, P. Li, J. L. Rohn, and B. Baum. Patch-basd within-objct classification. In Intrnational Confrnc on Computr Vision, 009. [] B. Brattoli, U. Buchlr, A.-S. Wahl, M. E. Schwab, and B. Ommr. Lstm slf-suprvision for dtaild bhavior analysis. In Computr Vision and Pattrn Rcognition, 07. [] U. Buchlr, B. Brattoli, and B. Ommr. Improving spatiotmporal slf-suprvision by dp rinforcmnt larning. In Europan Confrnc on Computr Vision, 08. [] M. Caron, P. Bojanowski, A. Joulin, and M. Douz. Dp clustring for unsuprvisd larning of visual faturs. In Europan Confrnc on Computr Vision, 08. [] O. Cick, A. Abdulkadir, S. S. Linkamp, T. Brox, and O. Ronnbrgr. d u-nt: larning dns volumtric sgmntation from spars annotation. In Intrnational Confrnc on Mdical Imag Computing and Computr-Assistd Intrvntion, 0. [] J. Dng, A. C. Brg, K. Li, and L. Fi-Fi. What dos classifying mor than 0,000 imag catgoris tll us? In Europan Confrnc on Computr Vision, 00. [7] J. Dng, W. Dong, R. Sochr, L. J. Li, K. Li, and L. Fi- Fi. Imagnt: A larg-scal hirarchical imag databas. In Computr Vision and Pattrn Rcognition, 009. [8] C. Dorsch, A. Gupta, and A. A. Efros. Unsuprvisd visual rprsntation larning by contxt prdiction. In Intrnational Confrnc on Computr Vision, 0. [9] J. Donahu, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzng, and T. Darrll. Dcaf: A dp convolutional activation fatur for gnric visual rcognition. In Intrnational Confrnc on Machin Larning, 0. [0] D. Erhan, Y. Bngio, A. Courvill, P. A. Manzagol, P. Vincnt, and S. Bngio. Why dos unsuprvisd pr-training hlp dp larning? Journal of Machin Larning Rsarch, (Fb): 0, 00. [] M. Evringham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zissrman. Th pascal visual objct classs (voc) challng. Intrnational Journal of Computr Vision, 88():0 8, 00. [] B. Frnando, H. Biln, E. Gavvs, and S. Gould. Slfsuprvisd vido rprsntation larning with odd-on-out ntworks. In Computr Vision and Pattrn Rcognition, 07. [] S. Gidaris, P. Singh, and N. Komodakis. Unsuprvisd rprsntation larning by prdicting imag rotations. In Intrnational Confrnc on Larning Rprsntations, 08. [] R. Girshick. Fast r-cnn. In Intrnational Confrnc on Computr Vision, 0. [] R. Girshick, J. Donahu, T. Darrll, and J. Malik. Rich fatur hirarchis for accurat objct dtction and smantic sgmntation. In Computr Vision and Pattrn Rcognition, 0. [] K. H, X. Zhang, S. Rn, and J. Sun. Dp rsidual larning for imag rcognition. In Computr Vision and Pattrn Rcognition, 0. [7] G. Huang, Z. Liu, K. Q. Winbrgr, and L. van dr Maatn. Dnsly connctd convolutional ntworks. In Computr Vision and Pattrn Rcognition, 07. [8] A. Joulin, L. van dr Maatn, A. Jabri, and N. Vasilach. Larning visual faturs from larg wakly suprvisd data. In Europan Confrnc on Computr Vision, 0. [9] A. Karpathy, G. Todrici, S. Shtty, T. Lung, R. Sukthankar, and L. Fi-Fi. Larg-scal vido classification with convolutional nural ntworks. In Computr Vision and Pattrn Rcognition, 0. [0] T. N. Kipf and M. Wlling. Smi-suprvisd classification with graph convolutional ntworks. In Intrnational Confrnc on Larning Rprsntations, 07. [] A. Krizhvsky, I. Sutskvr, and G. E. Hinton. Imagnt classification with dp convolutional nural ntworks. In Advancs in Nural Information Procssing Systms, 0. [] G. Larsson, M. Mair, and G. Shakhnarovich. Colorization as a proxy task for visual undrstanding. In Computr Vision and Pattrn Rcognition, 07. [] Q. V. L. Building high-lvl faturs using larg scal unsuprvisd larning. In Intrnational Confrnc on Spch and Signal Procssing, 0. [] Y. LCun, Y. Bngio, and G. E. Hinton. Dp larning. Natur, (7):, 0. [] H.-Y. L, J.-B. Huang, M. Singh, and M.-H. Yang. Unsuprvisd rprsntation larning by sorting squncs. In Intrnational Confrnc on Computr Vision, 07. [] Q. Li, J. Wang, D. Wipf, and Z. Tu. Fixd-point modl for structurd labling. In Intrnational Confrnc on Machin Larning, 0. [7] J. Long, E. Shlhamr, and T. Darrll. Fully convolutional ntworks for smantic sgmntation. In Computr Vision and Pattrn Rcognition, 0. [8] F. Milltari, N. Navab, and S. A. Ahmadi. V-nt: Fully convolutional nural ntworks for volumtric mdical imag sgmntation. In Intrnational Confrnc on D Vision, 0. [9] I. Misra, C. L. Zitnick, and M. Hbrt. Shuffl and larn: unsuprvisd larning using tmporal ordr vrification. In Europan Confrnc on Computr Vision, 0. [0] T. N. Mundhnk, D. Ho, and B. Y. Chn. Improvmnts to contxt basd slf-suprvisd larning. In Computr Vision and Pattrn Rcognition, 08. [] M. Noroozi and P. Favaro. Unsuprvisd larning of visual rprsntations by solving jigsaw puzzls. In Europan Confrnc on Computr Vision, 0. [] M. Noroozi, H. Pirsiavash, and P. Favaro. Rprsntation larning by larning to count. In Intrnational Confrnc on Computr Vision, 07. [] M. Noroozi, A. Vinjimoor, P. Favaro, and H. Pirsiavash. Boosting slf-suprvisd larning via knowldg transfr. In Computr Vision and Pattrn Rcognition, 08. [] G. Papandrou, L. C. Chn, K. Murphy, and A. L. Yuill. Wakly- and smi-suprvisd larning of a dcnn for smantic imag sgmntation. In Intrnational Confrnc on Computr Vision, 0. 9

10 [] S. Qiao, W. Shn, Z. Zhang, B. Wang, and A. L. Yuill. Dp co-training for smi-suprvisd imag rcognition. In Europan Confrnc on Computr Vision, 08. [] A. Radford, L. Mtz, and S. Chintala. Unsuprvisd rprsntation larning with dp convolutional gnrativ advrsarial ntworks. In Intrnational Confrnc on Larning Rprsntations, 0. [7] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn faturs off-th-shlf: an astounding baslin for rcognition. In Computr Vision and Pattrn Rcognition, 0. [8] S. Rn, K. H, R. Girshick, and J. Sun. Fastr r-cnn: Towards ral-tim objct dtction with rgion proposal ntworks. In Advancs in Nural Information Procssing Systms, 0. [9] Z. Rn and Y. J. L. Cross-domain slf-suprvisd multitask fatur larning using synthtic imagry. In Computr Vision and Pattrn Rcognition, 08. [0] O. Ronnbrgr, P. Fischr, and T. Brox. U-nt: Convolutional ntworks for biomdical imag sgmntation. In Intrnational Confrnc on Mdical Imag Computing and Computr-Assistd Intrvntion, 0. [] H. R. Roth, L. Lu, A. Farag, H. Shin, J. Liu, E. B. Turkby, and R. M. Summrs. Dporgan: Multi-lvl dp convolutional ntworks for automatd pancras sgmntation. In Intrnational Confrnc on Mdical Imag Computing and Computr-Assistd Intrvntion, 0. [] F. Roussau, P. A. Habas, and C. Studholm. A suprvisd patch-basd approach for human brain labling. IEEE transactions on mdical imaging, 0(0):8 8, 0. [] O. Russakovsky, J. Dng, H. Su, J. Kraus, S. Sathsh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Brnstin, t al. Imagnt larg scal visual rcognition challng. Intrnational Journal of Computr Vision, ():, 0. [] R. Santa Cruz, B. Frnando, A. Chrian, and S. Gould. Dpprmnt: Visual prmutation larning. In Computr Vision and Pattrn Rcognition, 07. [] K. Simonyan and A. Zissrman. Vry dp convolutional ntworks for larg-scal imag rcognition. In Intrnational Confrnc on Larning Rprsntations, 0. [] C. Szgdy, W. Liu, Y. Jia, P. Srmant, S. Rd, D. Angulov, D. Erhan, V. Vanhouck, A. Rabinovich, t al. Going dpr with convolutions. In Computr Vision and Pattrn Rcognition, 0. [7] X. Wang and A. Gupta. Unsuprvisd larning of visual rprsntations using vidos. In Intrnational Confrnc on Computr Vision, 0. [8] K. Q. Winbrgr and L. K. Saul. Unsuprvisd larning of imag manifolds by smidfinit programming. Intrnational Journal of Computr Vision, 70():77 90, 00. [9] Y. Xia, L. Xi, F. Liu, Z. Zhu, E. K. Fishman, and A. L. Yuill. Bridging th gap btwn d and d organ sgmntation with volumtric fusion nt. In Intrnational Confrnc on Mdical Imag Computing and Computr-Assistd Intrvntion, 08. [0] J. Yang, D. Parikh, and D. Batra. Joint unsuprvisd larning of dp rprsntations and imag clustrs. In Computr Vision and Pattrn Rcognition, 0. [] Q. Yu, L. Xi, Y. Wang, Y. Zhou, E. K. Fishman, and A. L. Yuill. Rcurrnt salincy transformation ntwork: Incorporating multi-stag visual cus for small organ sgmntation. In Computr Vision and Pattrn Rcognition, 08. [] Z. Zhang, C. Xi, J. Wang, L. Xi, and A. L. Yuill. Dpvoting: An xplainabl framwork for smantic part dtction undr partial occlusion. In Computr Vision and Pattrn Rcognition, 08. [] B. Zhou, A. Lapdriza, J. Xiao, A. Torralba, and A. Oliva. Larning dp faturs for scn rcognition using placs databas. In Advancs in Nural Information Procssing Systms, 0. [] J. Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaird imag-to-imag translation using cycl-consistnt advrsarial ntworks. In IEEE Intrnational Confrnc on Computr Vision, 07. 0