Search for New Resonance X W γ and Standard Model W γ Production Using Deep Learning Techniques

Transcription

1 Search for New Resonance X W γ and Standard Model W γ Production Using Deep Learning Techniques Wei Tang Duke University (ATLAS Collaboration) (Dated: April 17, 2018) The Standard Model is by far the most encompassing physics theory. With the recent discovery of the Higgs Boson, the Standard Model has performed extremely well when confronted with experimental data. However, the theory is intrinsically not complete. Extensions of the Standard Model predict new resonances decaying to a W boson and a photon. This thesis presents a search for such resonances produced in proton-proton collisions at s = 13 TeV using a dataset with an integrated luminosity of 36.1 fb 1 collected by the ATLAS detector at the Large Hadron Collider. The W bosons are identified through their hadronic decay channels. The data are found to be consistent with the expected background in the entire mass range investigated. Upper limits are set on the production cross section times decay branching ratio to W + γ of new resonances with mass between 1.0 and 6.8 TeV. The second part of this thesis looks at the extraction of the Standard Model W + γ production. The main background noise for the extraction comes from the Standard Model production of γ and jets. In order to streamline and improve the accuracy of the event filtering process, this thesis developed a deep neural net classifier to identify signal and background decay products and improved the signal to noise ratio. CONTENTS I. Introduction 2 II. Experiment Background 2 A. The Large Hadron Collider 2 B. The ATLAS Detector 2 III. Search for New X V γ Resonances 3 A. Simulation[3] 3 B. Hadronic Decay 4 C. X W/Z(q q) + γ Event Selections 5 D. Study of X V γ Selection Efficiencies at Parton-level 5 E. Cross Section Times Branching Ratio Upper Limits for X W γ 6 F. Study of W/Z Polarizations 6 IV. Standard Model W + γ and Background 8 A. Baseline and D2 Cuts 8 B. Background and Features 9 V. Neural Net 9 A. Methodology Random Structure Convergence Termination Metric of Performance Features Preprocessing 11 B. Training and Testing 12 Undergraduate thesis by Wei Tang. wt43@duke.edu; Physics Department, Duke University. C. Results Visualizations 13 VI. Hyper Parameter Tuning 14 A. Dynamic Learning Rate 15 B. Changing Layers of Neural Net 16 VII. Conclusions 17 Acknowledgements 17 References 17 VIII. Appendix 18 A. Neural Net Structure and Terminology 18 B. Dynamic Learning Rate 19 A. Exponential Decay 19 B. Stair Case Decay 19 C. Triangular 20 D. Stochastic Gradient Descent with Warm Restarts (SGDR) 20 C. Changing Layers of Neural Net 21 D. Tabulation of Neural Net Metric of Performances 23 A. 2-layer Neural Net 23 B. 3-layer Neural Net 24 C. 4-layer Neural Net 24 D. 5-layer Neural Net 24 E. Metric of Performances Plots 25 A. 2-layer Neural Net 25

2 2 B. 3-layer Neural Net 26 C. 4-layer Neural Net 28 D. 5-layer Neural Net 29 II. EXPERIMENT BACKGROUND A. The Large Hadron Collider I. INTRODUCTION This thesis first introduces background information about CERN, LHC and the ATLAS experiment in Section II. We then analyze proton-proton collision experimental data to search for evidence of new particles. Analysis of the data collected is divided into two parts. The first part is a summary of a talk presented at the 2017 American Physical Society Division of Particles and Fields conference. This research examines possible decay channels of an unknown massive boson X decaying into V plus a photon, where V can be either a W or a Z boson. The production mechanism can be either gluon-gluon fusion or q q annihilation. Figure 1 illustrates the possible production and decay channels. FIG. 1: X Vγ At CERN, the European Organization for Nuclear Research, physicists and engineers explore new fronts of particle physics. One of the major tasks is to look for unknown particles by locating new resonances. Such new resonances are located by analyzing data from two colliding proton beams. The Large Hadron Collider (LHC) is the latest addition to the CERN accelerator complex. Electromagnetic resonators accelerate proton beams to near the speed of light to collide with each other to produce p-p collisions with centre of mass energy s =13 TeV. The beams travel in opposite directions in separate beam pipes, which are two tubes kept at ultrahigh vacuum. The LHC consists of a 27-kilometer ring of superconducting magnets to direct proton beams. The superconducting magnets operate at an extremely low temperature of C, which requires liquid helium cooling systems all around. The LHC is not a perfect circle, but is made of eight arcs and eight insertions. The arcs contain the dipole directing magnets, with 154 in each arc. An insertion consists of a long straight section plus two (one at each end) transition regions. Thousands of magnets are used to direct and collimate proton beams to collide. The task requires astonishing precision that is comparable to firing two needles 10km apart and make them to collide heads-on. Section III describes preliminary results from a generic search for these new massive X bosons. The ultimate goal of the search is to explore the X mass range from 250 GeV to highest energy attained in 13 TeV p-p collision. Section III F explains the underlying reason for observed detector inefficiencies for spin 1 X W γ decays. The second part of the thesis attempts at superseding the traditional box cuts method employed in filtering signal events using machine learning classification techniques. Section IV describes the background and signal events we attempt to classify. Section V presents the classification performances using neural net. Sections VI A and VI B analyze the effects of dynamic learning rate techniques and changing hyper parameters. Last but not least, this thesis outlooks the future directions and discusses the strengths and weaknesses of the research. B. The ATLAS Detector There are two general purpose detectors at LHC, namely the ATLAS (A Toroidal LHC ApparatuS) and the CMS (Compact Muon Solenoid), probing p p and A A collisions. Figure 2 shows the overall ATLAS detector layout.

3 3 GHz, while the event data recording, based on technology and resource limitations, is limited to about 200 Hz. This requires an overall rejection factor of against minimum-bias processes while maintaining maximum efficiency for the new physics. The Level-1 (L1) trigger system uses a subset of the total detector information to make a decision on whether or not to continue processing an event, reducing the data rate to approximately 75 khz (limited by the bandwidth of the readout system, which is upgradeable to 100 khz). The subsequent two levels, collectively known as the high-level trigger, are the Level-2 (L2) trigger and the event filter. They provide the reduction to a final data-taking rate of approximately 200 Hz.[7] FIG. 2: Cut-away view of the ATLAS detector. The dimensions of the detector are 25 m in height and 44 m in length. The overall weight of the detector is approximately 7000 tonnes. The main sub-detector system comprises the inner detector, the electromagnetic calorimeters, the hadronic calorimeters, and the muon spectrometer. The nominal interaction point is defined as the origin of the coordinate system. Proton-proton beams travel in the direction defined to be the z-axis and the x-y plane is transverse to the direction. The azimuthal angle φ is measured around the beam axis in the x-y plane, and the polar angle θ is the angle from the beam axis. The pseudo-rapidity is defined as η = ln tan θ/2. In the case of massive W/Z boson or jets, the subject of this research, the rapidity y = 1/2 ln [(E + pz )/(E pz )] is used. The transverse momentum pt, the transverse energy ET are defined in the x-y plane, which are important parameters used in the research. The inner detector (ID) is designed to measure the trajectories of charged particles with great precision and to determine the position of the collision vertex and any secondary vertices. Secondary vertices are defined to be the point of decay of long-lived particles. The calorimeter system measures the energy of hadrons, electrons and photons. It provides coverage up to η = 4.9. Hadrons, electrons and photons deposit their energy during showering in the calorimeter system without reaching the muon system. The muon system surrounds the calorimeter system. It measures the trajectory of muons as they traverse the detector. The ATLAS detector is forward-backward symmetric with respect to the interaction point and covers a phase space of almost 4π steradians in solid angle around it. The magnet configuration comprises a thin superconducting solenoid surrounding the inner-detector cavity, and three large superconducting toroids (one barrel and two end-caps) around the calorimeters. The proton-proton interaction rate is approximately 1 III. SEARCH FOR NEW X V γ RESONANCES A. Simulation[3] Events considered for analysis are triggered on the presence of a photon candidate with pt greater than 140 GeV. If more than one photon or jet candidate are found, only the highest pt objects are considered further. The decrease of efficiency for resonances with lower mass is due to the kinematic thresholds of 200 GeV and 250 GeV required on the jet and photon pt respectively. The resonance type of interest to this study is the spin 1 X W γ q q + γ. An unknown massive boson X decays with a narrow width to a W boson and a photon. The narrow width chosen is 4 MeV. Reconstructed mass distributions for various signal hypotheses are shown in Figure 3. Peaks centered at the Z, W, and Higgs boson masses are clearly visible.

4 4 FIG. 4: Efficiencies for signal events to pass the baseline selection as a function of the resonance mass m X for different signal models, production, and spin hypothesis. The relevant line for this study is the spin 1 X q q X W γ. Full ATLAS detector simulation. B. Hadronic Decay FIG. 3: The mass distributions of large-r jets originating from Z, W, and H bosons resulting from decays of a resonance with a mass at m X =1 TeV and m X =4 TeV in samples of simulated events. The mass distribution of interest is the W γ hypothesis. The hadronic decays of the W/Z bosons have much higher branching ratio than the leptonic decays. The search sensitivity for X W/Z + γ can be improved by capturing a higher fraction of the W/Z bosons from their hadronic decays. The branching ratio of W/Z q q is around 70%. Figure 13 shows the production and decay channels. The first channel is spin 0 X produced via gluon-gluon fusion, decays to Z(q q)γ. The second channel is spin 1 X produced via q q annihilation, decays to W (q q)γ. The third channel is spin 2 X produced via gluon-gluon fusion, decays to Z(q q)γ. The spin 0 gluon-gluon fusion production with Zγ decay signals are generated using the POWHEG-BOX generator. Both spin 1 X W (q q) + γ and spin 2 X Z(q q) + γ signals are generated using the MadGraph generator. Figure 4 shows the total signal efficiency as a function of M X. The signal efficiency of all the selection cuts combined range from 45-65% for increasing M X. FIG. 5: Hadronic Decay Channel

5 5 C. X W/Z(q q) + γ Event Selections This search uses a selection of events based upon a photon trigger with transverse momentum P T > 140GeV, followed by identification of highly boosted W (q q) bosons using fat jets with cone size R = 1.0. A jet substructure variable D (β=1) 2 is used to identify fat jets with 2-jet substructure, to select hadronically decaying W q q bosons while suppressing jets from single quarks or gluons. Figure 6 shows the q q decays from a single W/Z boson. Due to the high momentum of the boson, the q q decay products are very close together and their subsequent shower merge. FIG. 7: P T γ > 250GeV Selection FIG. 6: Identification of W Boosted Jets The selection efficiency of the fat-jet identification of the W and Z bosons depends on the R(q q) decay distribution, and is therefore sensitive to the polarization of the W/Z bosons. This paper studies the signal efficiency as a function of the X boson mass using preliminary kinematic selection criteria. The kinematic selection criteria are: 1. η (q q) < 2 2. P T (q q) > 200GeV FIG. 8: η γ < 1.37 Selection 3. P T γ > 250GeV 4. η γ < 1.37 D. Study of X V γ Selection Efficiencies at Parton-level Figures 7 to 10 show the selection efficiencies for the preliminary kinematic cuts applied separately on the signal events. The examples shown here are for M X =4 TeV. The same selection cuts are applied to signal events of all M X slices. Figure 11 shows the total signal efficiencies from applying the combined kinematic selection cuts as a function of M X for the three different hadronic channels. The signal efficiencies increase abruptly from low M X to high masses. FIG. 9: P T q q > 200GeV Selection

6 = 1 TeV to 0.1 fb at M X = 6.8 TeV. There is not a significant deviation from the expected σ BR limits. 6 FIG. 10: η q q < 2 Selection FIG. 12: The 95% CL observed (solid line) and expected (dashed line) upper limits on σb for a spin = 1 resonance decaying to W γ as a function of the resonance mass. The green and yellow bands give the ±1 and ±2 standard deviations of expected limits. The local deviations of the observed from the expected upper limits on σb are a result of small deviations in the data from the best-fitting background-only model. FIG. 11: efficiency from combined kinematic selection cuts on all three production channels. 1. gluongluon fusion spin 0 X Z(q q)γ. 2. q q annihilation W (q q)γ. 3. gluon-gluon fusion spin 2 X Z(q q)γ. Parton-level only. E. Cross Section Times Branching Ratio Upper Limits for X W γ Upper limits on the product of cross section and branching ratio σ(pp X) BR(X W γ) are set using a profile likelihood method. Figure 12 shows the observed (solid lines) and median expected (dashed lines) 95% confidence level limits on the product of the production cross section times the branching ratio for the decay to a W boson and a photon of a boson X. The green and yellow band indicate 1 and 2 standard deviation intervals. A modified frequentist (CLs) method is used to set upper limits on the product, by identifying the value of σ BR for which CLs=0.05. The observed σ BR limits vary from 10 fb at M X FIG. 13: Hadronic Decay Channel F. Study of W/Z Polarizations The identification of W/Z boosted jet is affected by R(q q) distributions and hence the polarizations of the boson decay products. In order to quantify such decay, we define a helicity frame and analyzes the q q decay angular polarizations in the frame.

7 7 Figure 14 illustrates the definition of the helicity frame. The helicity frame is the W/Z boson rest frame with its Z axis defined to be along the same direction as the W/Z boson decay from the rest frame of massive boson X. We can reach the helicity frame by performing two consecutive sets of rotation and boost from the proton-proton rest frame. In the helicity frame, the q q decay products are back to back due to conservation of momentum. This paper studies the cos θ distribution to quantify the polarizations. (a) cos θ Distributions for purely longitudinal (blue) and purely transverse (red) polarizations FIG. 14: Helicity Frame Definition Theoretically, the cos θ distributions are different for the different W/Z production channels. Figure 15 shows the expected functional forms [1] of the decay angular distributions from Standard Model electroweak decay of the W boson. (b) cos θ Distributions for three channels at M X=5 TeV FIG. 16: Theoretical cos θ distributions and an example distribution The same fitting process is repeated for all M X slices and the B A parameters as a function of M X are plotted in Figure 17. Spin 0 and spin 2 productions are almost purely transversely polarized while the spin 1 production is almost purely longitudinally polarized. FIG. 15: Functional forms of decay polarizations for different W/Z production channels The cos θ q distributions are fitted to the functional forms of A + B cos 2 θ = A(1 + B A cos2 θ). The fitting parameter B A is hence used to quantify the polarization. B A = 1 indicates a purely transverse polarization whereas B A = 1 indicates a purely longitudinal polarization. Figure 16a illustrates the distributions for purely longitudinal and purely transverse polarizations. Figure 16b shows an example of cos θ distributions at M X =5 TeV. It clearly demonstrates that the spin 0 and spin 2 productions are much more transversely polarized, while the spin 1 production is much more longitudinally polarized. FIG. 17: B A polarization parameter for all M X We use fat jet selection to identify W/Z boosted jet. This is using a cone radius of R q q < 1, which is defined

8 8 to be R q q = η 2 q q + ϕ 2 q q (1) Figure 18 shows the R q q distributions for two M X at 1 TeV and 4 TeV. We obtain good containment on the parton level. However, real data has extra inefficiencies due to presence of jets. suppressed the initial stage γ radiation with a soft W production in figure 19c. By requiring that the γ has a transverse momentum larger than 250GeV, we suppressed the final stage γ radiation channel in figure 19d. The initial stage γ radiation production in figure 19b is still largely retained in signal events. Since real collision data will not be able to differentiate the two production channels, we do not attempt to clean up the signal any further. FIG. 18: R q q distributions for M X =1 TeV, 4 TeV (a) W γ triple gauge coupling (b) Initial stage γ radiation IV. STANDARD MODEL W + γ SIGNAL AND BACKGROUND The signal under study is the standard model W + γ production from pp collisions at s =13 TeV. The background is the standard model γ + jets production. The full data set is divided based on γ p T slices. The p T slice under study here is GeV. (c) Initial stage γ radiation with a soft W radiated off q (d) Final stage γ radiation FIG. 19: Four possible production channels present in the Monte Carlo simulation of pp W (q q)γ. A. Baseline and D2 Cuts For the W + γ events selected, there is a set of baseline γ + jet kinematic selection cuts applied as the first round of event selection. The baseline selections are: 1. Use the detector region with the best γ detection: P t γ > 250GeV, η γ < Use the detector region with the best jet detection: P t jet > 200GeV, η jet < 2 D2 is a variable specifically designed to differentiate single jet showers from di-jet showers. As illustrated in Figure 20, the D2 algorithm [3] takes a unit radius circle in the ϕ, η space of jet showers and calculates the variable as: 3. Remove photon tracks disguised as jets: R γ Jet > 1, where R γ Jet = η 2 γ Jet + ϕ2 γ Jet D 2 = ECF 3 ECF 3 1 ECF 3 2 (2) There are four types of production channels in the standard model W + γ production, shown in Figure 19. The standard model treats the W boson as a point particle. Hence we are interested in the W γ triple gauge coupling events to probe composite structures (if any) of the W boson. Ideally, signal events under study should purely come from the production channel shown in figure 19a. However, the standard model production produces all four production channels. By requiring that the W boson has a transverse momentum larger than 150GeV, we Where the ECF variables are the energy correlation functions, representing the likelihood of a jet shower containing some number of jets. The D2 distributions of the signal and background events are shown in Figure 21. The ECF 2 variable could be 0, in which case we designate the D2 to be -1. The D2 variable cut is implemented as a function of the combined mass corrected transverse momentum of this jet. Figure 22 shows the changing D2 upper cut limit as the transverse momentum changes.

9 mass is apparently the most important feature to retain. Figure 23 shows the jet mass of the highest pt jet in the standard model γ + jets decay background. Figure 24 shows the jet mass in the W + γ signal, which shows a clear peak near the mass of W. 9 FIG. 20: D2 algorithm illustration. FIG. 23: Jet mass of highest Pt jet in background. (a) (b) Background FIG. 21: D2 distributions in signal and background. The signal data set is standard model production of W + γ, with the p T of γ in between 280 to 500 GeV. The background data set is the standard model production of γ + jets, in the same γ p T slice. FIG. 24: Jet mass of highest Pt jet in signal. The higher peak shows W (q q) invariant mass. FIG. 22: D2 cut is a function of the jet p T. There are 21 event features in total that we took into account to develop the multi variant analysis method to distinguish the W (q q) + γ signal from γ+ jets background. The features are: 1. Pt, η, ϕ, mass, D 2, and number of sub-jets for the 3 highest Pt jets. 2. Pt, η, ϕ of the highest Pt γ. V. NEURAL NET B. Background and Features The important evidence that any classification mechanics works is whether it could successfully retain the signal features while suppressing the background features. In our search for new resonances, the jet invariant This study uses artificial neural net as a classifier to filter the standard model production of W + γ from γ + jets events. Appendix A introduces the general structure and terminology often used with neural net. The following sections outline the specific methodology involved in this study.

10 10 A. Methodology 1. Random Structure The input layer always has the same number of neurons as the number of features, while the output layer always has the same number of neurons as the number of classes intended, which is 2 in our case. However, different neural net structures of the hidden layers greatly affect the performances. In order to look for a good structure, I implemented a random structure test. Empirically, neural nets with an inverse pyramid structure tends to have better performances. An inverse pyramid structure has more neurons in the upper layers and the number decreases as the layer goes deeper. The maximum number of neurons is restricted by the equation N h = N s α (N i + N o ) (3) where N s is the number of training samples, N i is the number of features and N o is the number of output classes. α is a user defined variable that aims at preventing overfitting. In my random structure tests, α = 8. The number of neurons in each layer is randomly chosen, with their ranges contained in between two normal distributions. The first normal distribution has µ = 1 and σ = n 2, with its pdf when layer is 1 scaled to N h. The second normal distribution also has µ = 1 and σ = n 2, with its pdf when layer is 1 scaled to N h 2. The number of neurons in each layer takes a random number in between the two normal distributions probability density functions. Figure 25 shows an example for a 3-layer neural net. the first layer is N s N h = α (N i + N o ) = 8 (27 + 2) 388 Hence, hidden layer one has a random number of nodes in between 194 and 388, hidden layer two is in between 155 and 310, hidden layer 3 is in between 79 and 159. Since examining each layer nodes combination is computationally impossible, I tested 10 such random structures for 2-, 3-, 4- and 5-layer neural net respectively. 2. Convergence Termination The training samples are fed in to the neural in batches of 1000 samples each. After the neural net sees all the events for once, it completes a so-called epoch. To converge to a minimum in loss, it usually requires more than just one epochs. I use the softmax cross entropy function as the loss of the neural net. The cross-entropy error function over a batch of multiple samples of size n, denoted as T and its corresponding ground truth labels denoted as Y can be calculated as: ζ(t, Y ) = n n ζ(t i, y i ) = i=1 i=1 c=1 2 t ic log y ic (4) Where t ic is 1 if and only if sample i belongs to class c, and y ic is the output probability that sample i belongs to class c. In our study, there are only two classes of signal and background. To reduce unnecessary rounds of training, the training process terminates when the change in loss is less than 1% than the previous epoch. It is possible that the neural net fails to find a local minimum and never converges during training. Hence the max number of epochs to be trained is manually set to 50. According to observation, the neural net never reaches near 50 epochs. 3. Metric of Performance FIG. 25: Range of random number of neurons in each layer for a 3-layer neural net. The vertical axis is the number of neurons and the horizontal axis is the depth of layers. In this example, the maximum number of neurons in To quantize how well a certain neural net is at our classification work, I employed two metrics. The first is called the Receiver Operating Characteristic (ROC) curve. Figure 26 shows an example plot of a ROC curve. The vertical axis is the background rejection ratio, calculated as the percentage of background events rejected by the neural net. The horizontal axis is the signal efficiency, calculated as the percentage of signal retained by the neural net. The ROC curve is obtained by gradually varying the possibility threshold cut on the final probability output from the neural net. Higher area under

11 11 curve (AUC) means better performance. AUC=0.5 indicates that the neural net performs no better than random guessing. Figure 26 shows 5 ROC curves for 5 different 3- layer neural net structures and their corresponding AUC values. 4. Features Preprocessing We have altogether 21 features for each event to form our feature space. However, since photon jets removal removes some jets from events, it is sometimes the case that an event does not even have 3 jets. In that case, we manually assign pseudo feature parameters that are clearly outside the range of distributions of real jets. The reason to do this is because our multi variant method demands consistent features in all events. Figure 28 demonstrates a manual addition of fake jets. In our signal events, there is a large number of events without even a second jet. In those cases, we manually assign an η value of -5, which is definitely outside the range of η distributions of the real jets. By having manual fake jets addition, we provide the possibility that the different number of jets in events could also be indirectly utilized as a differentiation parameter, since the fake jets greatly distort the feature distributions. FIG. 26: ROC curves for 5 different 3-layer neural net using all features and constant learning The second metric of performance is called the signal over background plot. Figure 27 shows an example. The plus 1 in the denominator prevents having a division by 0 error. Obviously, higher ratio at some probability threshold indicates better filtering. In the example plot, a probability threshold cut at roughly 0.25 provides the best ratio of 30. FIG. 28: Manual fake jets addition. When an event does not have a second jet available, a fake η = 5 value is used instead. This value is chosen because it is clearly outside the range of true jets η. Similar process is used for all 18 features that are associated with jets. Manual fake entries are not necessary for the γ features since there is always a photon. FIG. 27: over background ratio curves for 3-layer neural net using all features and constant learning Besides manual fake jets addition technique, we recognize that different features vary greatly in ranges. For example, the η variable typically is in the range of ±4, while the mass of jets could go as high as 200GeV. However, the huge difference in ranges does not mean that one feature is much more important than the other. Our assumption is that all features should be treated equally. Therefore we also adopted the feature normalization technique. For each feature column, we find the maximum and minimum feature and normalize the features as: x i,j = x i,j min j (5) max j min j where x i,j is the j th feature in the i th event. And

12 12 x i,j is the normalized feature, min j and max j are the extremums of the j th feature across all events, including both the signal and the background events. The normalization procedure preserves the distribution of features and restrict the range of values of all features to be from 0-1. This prevents our classification structure to be misled into thinking that mass of jet is million times more important than η of jets. Due to similar reasoning, I label the signal and background events using 2d vectors instead of scalar labels. A one dimensional scalar labelling insinuates that there exists some quantitative relationship between a signal and a background event. For example, labelling background as 1 and signal as 2 indicates that a signal event is somehow 2 times larger than a background event, which does not carry any physical meaning. Instead, I labeled the background events as (0,1) and signal events as (1,0). Orthogonal vector labelling erases hints at quantitative relationships. (a) Mass of highest Pt jet, all features B. Training and Testing and background events are mixed with a 1:10 ratio, which is a realistic approximation to actual collision data. The mixed data is further divided into training and testing sets. 90% of the mixed events are randomly chosen to be the training set, the remaining 10% is reserved for testing set. The important metric to test the performance of the classification is to look at the test set jet mass distributions. The mixed signal and background data this study looks at is divided into three categories. The first category uses all 21 features as described in section IV B, preprocessed as described in section V A 4, and is labeled as all. (b) Mass of highest Pt jet, high level features The second category uses only the 9 high level features, which comprise mass, D2, and number of sub-jets for the 3 highest Pt jets. This category is labeled as high level. The last category uses all the 20 features except the D2 variable, and is labeled as no D2. Since each category draws its test set randomly, the test set jet mass distributions vary to a small extent. Figure 29 shows the mass of the highest Pt jet distribution from the random test samples in all three categories. The area under graphs have been normalized to 1. (c) Mass of highest Pt jet, absent D2 FIG. 29: Unfiltered leading Pt jet mass distributions in three categories. Since the testing sets are drawn at random from the mixed events, the distributions are quite similar, though not identical. The signal to background ratio is approximately 1:10, which makes the W (q q) mass peak largely insignificant. (a) shows the unfiltered mass distributions picked for the all features category. (b) shows that for the high level features category and (c) shows that for the no-d2 category.

13 13 C. Results Visualizations This section presents the filtering results by using all three categories of features and the constant learning Various neural net structures are used. Figure 30 shows simultaneously the unfiltered jet mass and filtered jet mass using all the features and has a neural net structure of [924, 561, 331]. Figures 31 and 32 show the same plots using only the high level features and omitting the D2 variable. All three categories show an enhanced peak near the mass of W as compared to the unfiltered mass. FIG. 32: Mass of highest Pt jet, absent D2. The neural net structures chosen have the highest area under curve of ROC curves, among the various structures tested. FIG. 30: Mass of highest Pt jet, all features. The neural net structures chosen have the highest area under curve of ROC curves, among the various structures tested. To gauge the performance of three categories, Figures 33, 34 and 35 show the Background+1 ratios. The neural net structure with the highest AUC values in each category are the blue line respectively. The highest ratio is almost the same for both all feature and absent D2 feature sets categories, both are higher than using only the high level features. This indicates that a 3-layer neural net is not able to infer information carried by low level features from only high level features. Using a more complete set of features is helpful to enhance the filtering performance of neural nets. FIG. 31: Mass of highest Pt jet, high level features. The neural net structures chosen have the highest area under curve of ROC curves, among the various structures tested. FIG. 33: Over Background Ratio, all features.

14 14 (b) Evolution of the cross entropy loss, using absent-d2 (a) Evolution of the cross entropy loss, using all features and features and constant learning constant learning FIG. 34: Over Background Ratio, high level features. FIG. 36: Evolution of loss as a function of epochs. First run. This phenomenon of more required epochs till convergence diminished in a more recent round of training. Figures 37a and 37b show the same quantity plot. The only condition changed in between these two figures and figures 36a and 36b is that the initialization of the neural nets parameters may be different. The comparison shows that the performance is unstable and depends on initialization of the parameters. (b) Evolution of the cross entropy loss, using absent-d2 (a) Evolution of the cross entropy loss, using all features and features and constant learning constant learning FIG. 35: Over Background Ratio, no D2 features. FIG. 37: Evolution of loss as a function of epochs. Second run. While there are no significant ratio differences between the all-feature and absent-d2 categories, figures 36a and 36b show that the two require quite different number of epochs to converge. Without the D2 variable, the neural net takes almost double the number of epochs to converge, although eventually reaching about the same loss. The evidence hence shows that all-feature category requires the least calculation and provides the best filtering. VI. HYPER PARAMETER TUNING An artificial neural net has many hyper parameters associated with it that can be tuned to achieve better performance. This section looks at two of those hyper parameters, learning rate and depths of neural net. Section VI A outlines several learning rate models we used. While section VI B analyzes the effects of using different depths of neural nets.

15 15 A. Dynamic Learning Rate The previous section describes neural net training using a constant learning While being simple, it suffers from the possibility that the training does not reach the bottom of the minimum. To improve accuracy when nearing the minimum, I adopted two dynamic learning rate models, the stair case decay and the exponential decay learning rates. Both learning rate models decay the learning rate as training proceeds. Figure 38 shows an example of an exponentially decayed learning rate as a function of global step. Global step is defined to be the number of small batches trained. The decay is calculated as global step LR t = LR 0 a T (6) FIG. 39: Stair case learning It is similar to exponentially decayed learning The difference is that staircase learning rate maintains a constant learning rate in certain integer divisions of global steps, rather than decaying smoothly. Where LR t is the dynamic learning rate, LR 0 is the base learning rate, a is a decay rate constant and T is decay width constant. I set LR 0 = 0.001, a = 0.1 and decay width to be one epoch. Both the exponential decay and staircase decay aim at enhancing accuracy when nearing a local minimum. One of the major problems that neural net faces is that the training may be trapped in a local minimum, failing to recognize a global minimum. The following two types of oscillatory learning rate models target this issue. Figure 40 shows an example of a triangular learning rate oscillation. The periodic oscillations in learning rate helps the training jump out of a local minimum and look for a potentially better global minimum in convergence. FIG. 38: Exponentially decayed learning Stair case decay is quite similar with exponential decay. global step The difference is that decaywidth is an integer division and the decayed learning rate follows a staircase function. Figure 39 shows an example of a staircase decay learning FIG. 40: Triangular learning The rise and fall width are both taken to be one epoch. The stochastic gradient descent with warm restarts (SGDR) adopts a cosine decay with a restart. The learn-

16 16 ing rate LR t is given by: LR t = 1 2 LR 0 [1 + cos(πt)] (7) Where t is the phase of the current global step, given by: t = global step % decay width decay width (8) The decay width is set to be 2 epochs in the study. When global step is multiples of the decay width, learning rate jumps back to the base learning rate, which is set to be Figure 41 shows an example of such a learning rate model. Learning Rate Highest AUC Background+1 Converged Loss SGDR exp constant staircase triangular Learning Rate TABLE I: 3-layer Neural Net, all features Highest AUC Background+1 Converged Loss triangular exp SGDR staircase constant Number of Epochs Number of Epochs TABLE II: 3-layer Neural Net, high level features FIG. 41: SGDR learning The metrics of performance plots using these dynamic learning rates are included in Appendix B. Tables I, II and III detail the metric of performances results from using a 3-layer neural net for all three categories of feature sets. The SGDR dynamic learning rate schedule with all feature category provides the highest AUC performance among all learning rate models and feature sets. SGDR also provides the highest Background+1 ratio using no-d2 feature set. It also gives the lowest converged loss when using the all feature category. Hence SGDR learning rate model out performs the other learning rate schedules. However, it also requires more epochs to converge in the all feature category, indicating higher calculation costs. It is noteworthy that using only the high level features provides worse performance in all metrics, using any learning rate schedules. This could be attributed to that the neural net is not able to infer information carried by low level features (the 4-vectors) from just the high level features. Learning Rate Highest AUC Background+1 Converged Loss staircase exp triangular constant SGDR Number of Epochs TABLE III: 3-layer Neural Net, no-d2 features B. Changing Layers of Neural Net The previous sections show results using a 3-layer neural net. This section looks at the effect of different number of layers on our metrics of performance. 2-, 3-, 4-, and 5-layer neural nets are used on all three categories of features and using all five learning rate schedules. Appendix C contains the detailed results obtained from such hyper parameter tuning. The cross entropy loss clearly shows a decreasing trend as the neural nets become deeper for all five learning rate schedules. The signal over background ratio generally increases as neural nets become deeper. Both the exponentially decayed and staircase decayed learning rate schedules suffers from the possibility that they may fall into the same local minimum during training. And using deeper layers of neural nets does not help get out of it. As a contrast, the SGDR learning rate schedule requires more epochs to reach convergence as neural nets

17 17 become deeper, as expected. In addition, the SGDR learning rate generally gives the lowest loss and the highest signal over background ratio in all depth of neural nets. Hence, it justifies the more epochs required to reach convergence. In fact, the two oscillation learning rate schedules out perform the non-oscillation learning rate schedules as they reach lower loss and have higher signal over background ratio. Using only the high-level features makes the signal over background ratio more chaotic than using all features. And there is no easily identifiable trend in this category of features. In addition, the SGDR learning rate schedule is no longer the best model. On the contrary, the non-oscillation learning rate schedules perform better when only using the high level features. However, using only the high level features generally incurs higher loss and lower signal over background ratios than the other two categories. This finding confirms again that even deeper structures are not able to infer the lost information carried by the 4-vectors of particles, although deeper structure still offers better performance when compared within the category. A comparison of using the all feature and the no- D2 feature categories shows that removing the D2 variable has little effect in converged loss and signal over background ratio obtained. Removing the D2 variable incurs a higher training cost in most of the cases. VII. CONCLUSIONS This thesis looks at proton-proton collision at s=13 TeV data from LHC at CERN. Upper limits are set on the production cross section times decay branching ratio to W + γ of new resonances with mass between 1.0 and 6.8 TeV. Comparison with the expected standard model theory predictions does not reveal statistically significant enough deviations. It hence concludes that the current search does not suggest evidence of new resonances. The deep learning technique of neural nets is used to improve filtering of the standard model W + γ production from the standard model γ + jets production background. We analyzed various neural net structures, learning rate models and different sets of features. We conclude that the SGDR learning rate provides better overall performance and deeper network structures tend to perform better. However, the cost of calculation also increases drastically as we aim for higher filtering accuracy. ACKNOWLEDGEMENTS I thank James B. Duke Professor of Physics, Alfred Goshaw, for his invaluable support and guidance for this thesis research. His insights and expertise guided this thesis project. I also thank PhD of Physics, Minyu Feng, for his assistance with the data selection methodology and other general assistance. I would also like to show gratitude to PhD of Physics, Sourav Sen, for sharing his knowledge in deep learning techniques. [1] arxiv:hep-ph/ [2] arxiv: [hep-ph] [3] HIGG [4] arxiv: v5 [cs.lg] [5] Jordan, Jeremy. Setting the learning rate of your neural network. Jeremy s Blog, Jeremy s Blog, 3 Mar. 2018, [6] CERN Accelerating science. (n.d.). Retrieved March 19, 2018, from [7] The ATLAS Collaboration et al 2008 JINST 3 S08003

18 18 Appendix A VIII. APPENDIX NEURAL NET STRUCTURE AND TERMINOLOGY An artificial neural net is widely used as a classification structure. The structure and working principles of a neural net is illustrated in Figure 42. A basic neural net comprises of three parts, the input layer, the hidden layers and the output layer. Each layer has a certain number of computation units called neurons. Each neuron is nothing more than a matrix multiplication and an addition. Upon taking an input vector, each neuron first calculates a weighted sum of the vector by multiplying the input vector by a weight vector according to This vector then undergoes a function called the activation function. There are many models of activation functions, we use the sigmoid function as shown in Figure 43. In the sigmoid activation function, the final output from a single neuron is computed as S(a) = ea e a + 1 (11) Hence the output from a single neuron is always restrained to be between 0-1. a = w T x (9) where w is the weight vector and x is the input feature vector. The scalar product a is then offset by a scalar b. Therefore, the final output from each neuron is a = w T x + b (10) Both the weight vector and the offset constant are parameters obtained from the training of the neural net. The input layer neurons each has a dimension of #features. The neurons on the i th hidden layer each has dimensions of #neurons on the (i 1) th layer. The output layer has one neuron with dimensions of #neurons on the last hidden layer #classes. #classes is the number of categories the neural net is trying to categorize into. Therefore, the overall input into a neural net is the feature vector, and the final output is a vector of dimension #classes. The component of the output vector indicates the probability of a particular feature vector belonging to a class. Take the neural net structure in Figure 42 as an example, the input vector to the input layer neurons is the feature vector, in our case, it has 21 features hence a 21 dimension vector. Each one of the three neurons in the input layer then calculates a different scalar output, based on what weight vector and offset constant they have. FIG. 43: Sigmoid activation function. The result from the input layer is hence three scalars, one from each neuron. This 3-d vector is then fed into each one of the four neurons in hidden layer 1. After the same process of multiplication, offset and activation function, the result from the hidden layer 1 is hence a 4-d vector. This is further fed into hidden layer 2 to produce another 4-d vector. The output layer eventually computes a scalar product of this 4-d vector and offset it to obtain a 2-d vector, which becomes the output of this neural net example. Take note that the application of activation function does not happen for the output layer. For a two-class classification problem such as ours, the output from the output layer is a 2-d vector, with each component indicating the possibility of a particular event being either a signal or background. FIG. 42: Artificial neural net structure. During training, the training samples are fed in as batches of 1000 events, instead of one by one. Upon seeing each batch of 1000 events, the neural net updates its weighs and biases based on the Adam Optimizer by Tensorflow to optimize accuracy. The learning rate for the optimizer is default to be a constant, set at

19 19 Appendix B DYNAMIC LEARNING RATE B Stair Case Decay A Exponential Decay (a) Filtered mass. (a) Filtered mass. (b) over noise ratio. (b) over noise ratio. (c) ROC curves. (c) ROC curves. FIG. 44: Metric of performance for 3-layer neural net, using all features and an exponential decay learning FIG. 45: Metric of performance for 3-layer neural net, using all features and a staircase decay learning

20 20 C Triangular D Stochastic Gradient Descent with Warm Restarts (SGDR) (a) Filtered mass. (a) Filtered mass. (b) over noise ratio. (b) over noise ratio. (c) ROC curves. FIG. 46: Metric of performance for 3-layer neural net, using all features and a triangular oscillation learning (c) ROC curves. FIG. 47: Metric of performance for 3-layer neural net, using all features and an SGDR oscillation learning

21 21 Appendix C CHANGING LAYERS OF NEURAL NET This section provides more in-detailed analysis of the effect of different number of layers on our metrics of performance. Figure 48 shows the number of epochs till convergence, the cross entropy loss and the Background+1 ratio as a function of different number of layers in neural nets. The cross entropy loss clearly shows a decreasing trend as the neural nets become deeper for all five learning rate schedules. The signal over background ratio generally increases as neural nets become deeper. One anomaly happens for constant learning rate schedule, where it sees a decrease from 4-layer to 5-layer neural net. within the category. Figure 50 shows the metrics as a function of neural net depth when using the no-d2 category of features. Again, the decreasing trend of the cross entropy loss happens for all five learning rate schedules. Figure 50a shows that the two oscillation schedules require more epochs to converge than the two decay schedules. One difference with the all-feature category is that the constant learning rate requires the most number of epochs in all depths of neural nets. Figure 48a shows an interesting observation. The umber of epochs required till convergence stays the same for exponentially decayed and staircase decayed learning rate schedules, always at 7. This indicates that both learning rate schedules may have fallen into the same local minimum during training. And using deeper layers of neural nets does not help get out of it. As a contrast, the SGDR learning rate schedule requires more epochs to reach convergence as neural nets become deeper, as expected. Figure 48b and 48c show that the SGDR learning rate generally gives the lowest loss and the highest signal over background ratio in all depth of neural nets. Hence, it justifies the more epochs required to reach convergence. In fact, the two oscillation learning rate schedules out perform the non-oscillation learning rate schedules as they reach lower loss and have higher signal over background ratio. The only exception is that the most simplistic constant learning rate achieves better end results than the triangular learning rate schedule in the range of depths analyzed under this study. Figure 49 shows the metrics as a function of neural net depth when using only the high level features. The decreasing trend of the cross entropy loss is still true. However, the signal over background ratio becomes more chaotic than using all features. While exponential decay, staircase decay and constant learning rate schedules reach higher ratio when the depth increases, it is not true for the two oscillation learning rate schedules. In addition, the SGDR learning rate schedule is no longer the best model. On the contrary, the nonoscillation learning rate schedules perform better when only using the high level features. However, using only the high level features generally incurs higher loss and lower signal over background ratios than the other two categories. This finding confirms again that even deeper structures are not able to infer the lost information carried by the 4-vectors of particles, although deeper structure still offers better performance when compared