Bypassing Space Explosion in High-Speed Regular Expression Matching

Transcription

1 Bypassing Spae Explosion in High-Speed Regular Expression Mathing Jignesh Patel Alex X. Liu Eri Torng Astrat Network intrusion detetion and prevention systems ommonly use regular expression (RE) signatures to represent individual seurity threats. While the orresponding DFA for any one RE is typially small, the DFA that orresponds to the entire set of REs is usually too large to e onstruted or deployed. To address this issue, a variety of alternative automata implementations that ompress the size of the final automaton have een proposed suh as XFA and D 2 FA. The resulting final automata are typially muh smaller than the orresponding DFA. However, the previously proposed automata onstrution algorithms do suffer from some drawaks. First, most employ a Union then Minimize framework where the automata for eah RE are first joined efore minimization ours. This leads to an expensive NFA to DFA suset onstrution on a relatively large NFA. Seond, most onstrut the orresponding large DFA as an intermediate step. In some ases, this DFA is so large that the final automaton annot e onstruted even though the final automaton is small enough to e deployed. In this paper, we propose a Minimize then Union framework for onstruting ompat alternative automata fousing on the D 2 FA. We show that we an onstrut an almost optimal final D 2 FA with small intermediate parsers. The key to our approah is a spae and time effiient routine for merging two ompat D 2 FA into a ompat D 2 FA. In our experiments, our algorithm runs on average 155 times faster and uses 1500 times less memory than previous algorithms. For example, we are ale to onstrut a D 2 FA with over 80,000,000 states using only 1GB of main memory in only 77 minutes. I. INTRODUCTION A. Bakground and Prolem Statement The ore omponent of today s network seurity devies suh as Network Intrusion Detetion and Prevention Systems is signature ased deep paket inspetion. The ontents of every paket need to e ompared against a set of signatures. Appliation level signature analysis an also e used for deteting peer-to-peer traffi, providing advaned QoS mehanisms. In the past, the signatures were speified as simple strings. Today, most deep paket inspetion engines suh as Snort [1], [2], Bro [3], TippingPoint X505 and Ciso seurity applianes use regular expressions (REs) to define Alex X. Liu is the orresponding author of this paper. alexliu@se.msu.edu. Tel: +1 (517) Fax: +1 (517) Jignesh Patel and Eri Torng are with the Department of Computer Siene and Engineering, Mihigan State University, East Lansing, MI, Alex X. Liu is with the Department of Computer Siene and Tehnology at Nanjing University and the Department of Computer Siene and Engineering at Mihigan State University. The preliminary version of this paper titled Bypassing spae explosion in regular expression mathing for network intrusion detetion and prevention systems was pulished in the proeedings of the Network and Distriuted System Seurity Symposium (NDSS), This work is supported in part y the National Siene Foundation under Grant Numer CNS {patelji1, alexliu, torng}@se.msu.edu Manusript reeived Septemer 16, the signatures. REs are used instead of simple string patterns eause REs are fundamentally more expressive and thus are ale to desrie a wider variety of attak signatures [4]. As a result, there has een a lot of reent work on implementing high speed RE parsers for network appliations. Most RE parsers use some variant of the Deterministi Finite State Automata (DFA) representation of REs. A DFA is defined as a 5-tuple (Q,Σ,δ,q 0,A), where Q is a set of states, Σ is an alphaet, δ: Q Σ Q is the transition funtion, q 0 Q is the start, and A Q is the set of aepting states. Any set of REs an e onverted into an equivalent DFA with the minimum numer of states [5], [6]. DFAs have the property of needing onstant memory aess per input symol, and hene result in preditale and fast andwidth. The main prolem with DFAs is spae explosion: a huge amount of memory is needed to store the transition funtion whih has Q Σ entries. Speifially, the numer of states an e very large (state explosion), and the numer of transitions per state is large ( Σ ). To address the DFA spae explosion prolem, a variety of DFA variants have een proposed that require muh less memory than DFAs to store. For example, there is the Delayed Input DFA (D 2 FA) proposed y Kumar et al. [7]. The asi idea of D 2 FA is that in a typial DFA for real world RE set, given two states u and v, δ(u,) = δ(v,) for many symols Σ. We an remove all the transitions for v from δ for whih δ(u,) = δ(v,) and make a note that v s transitions were removed ased onu s transitions. When the D 2 FA is later proessing input and is in state v and enounters input symol x, if δ(v,x) is missing, the D 2 FA an use δ(u,x) to determine the next state. We an do the same thing for most states in the DFA, and it results in tremendous transition ompression. Kumar et al. oserve an average derease of 97.6% in the amount of memory required to store a D 2 FA when ompared to its orresponding DFA. In more detail, to uild a D 2 FA from a DFA, just do the following two steps. First, for eah stateu Q, pik a deferred state, denoted y F(u). (We an have F(u) = u.) Seond, for eah state u Q for whih F(u) u, remove all the transitions for u for whih δ(u,x) = δ(f(u),x). When traversing the D 2 FA, if on urrent state u and urrent input symol x, δ(u,x) is missing (i.e. has een removed), we an use δ(f(u), x) to get the next state. Of ourse, δ(f(u),x) might e missing too, in whih ase we then use δ(f(f(u)), x) to get the next state, and so on. The only restrition on seleting deferred states is that the funtion F annot reate a yle other than a self-loop on the states; otherwise all states on that yle might have their transitions

2 2 on some x Σ removed and there would no way of finding the next state. Figure 1(a) shows a DFA for the REs set {.*a.*,.*.*}, and Figure 1() shows the D 2 FA uilt from the DFA. The dashed lines represent deferred states. The DFA has = 3328 transitions, whereas the D 2 FA only has 1030 atual transitions and 9 deferred transitions. D 2 FA are very effetive at dealing with the DFA spae explosion prolem. In partiular, D 2 FA exhiit tremendous transition ompression reduing the size of the DFA y a huge fator; this makes D 2 FA muh more pratial for a software implementation of RE mathing than DFAs. D 2 FAs are also used as starting point for advaned tehniques like those in [8], [9]. This leads us to the fundamental prolem we address in this paper. Given as input a set of REs R, uild a ompat D 2 FA as effiiently as possile that also supports frequent updates. Effiieny is important as urrent methods for onstruting D 2 FA may e so expensive in oth time and spae that they may not e ale to onstrut the final D 2 FA even if the D 2 FA is small enough to e deployed in networking devies that have limited omputing resoures. Suh issues eome douly important when we onsider the issue of the frequent updates (typially additions) to R that our as new seurity threats are identified. The limited resoure networking devie must e ale to effiiently ompute the new D 2 FA. One sutle ut important point aout this prolem is that the resulting D 2 FA must report whih RE (or REs) from R mathed a given input; this applies eause eah RE typially orresponds to a unique seurity threat. Finally, while we fous on D 2 FA in this paper, we elieve that our tehniques an e generalized to other ompat RE mathing automata solutions [10] [14]. B. Summary of Prior Art Given the input RE set R, any solution that uilds a D 2 FA for R will have to do the following two operations: (a) union the automata orresponding to eah RE in R and () minimize the automata, oth in terms of the numer of states and the numer of edges. Previous solutions [7], [15] employ a Union then Minimize framework where they first uild automata for eah RE within R, then perform union operations on these automata to arrive at one omined automaton for all the REs in R, and only then minimize the resulting omined automaton. In partiular, previous solutions typially perform a omputationally expensive NFA to DFA suset onstrution followed y or omposed with DFA minimization (for states) and D 2 FA minimization (for edges). Consider the D 2 FA onstrution algorithm proposed y Kumar et al. [7]. They first apply the Union then Minimize framework to produe a DFA that orresponds to R and then onstrut the orresponding minimum state DFA. Next, in order to maximize transition ompression, they solve a maximum weight spanning tree prolem on the following weighted graph whih they all a Spae Redution Graph (SRG). The SRG has DFA states as its verties. The SRG is a omplete graph with the weight of an edge w(u,v) equal to the numer of ommon transitions etween DFA states u and v. One the spanning tree is seleted, a root state is piked and all edges are direted towards the root. These direted edges give the deferred state for eah state. Figure 1() shows the SRG uilt for the DFA in Figure 1(a). Behi and Crowley also use the Union then Minimize Framework to arrive at a minimum state DFA [15]. At this point, rather than using an SRG to set deferment states for eah state, Behi and Crowley use state levels where the level of a DFA state u is the length of the shortest string that takes the DFA from the start state to state u. Behi and Crowley oserved that if all states defer to a state that is at a lower level than itself, then the deferment funtion F an never produe a yle. Furthermore, when proessing any input string of length n, at most n 1 deferred transitions will e proessed. Thus, for eah state u, among all the states at a lower level than u, Behi and Crowley set F(u) to e the state whih shares the most transitions with u. The resulting D 2 FA typially has a few more transitions than the minimal D 2 FA that results y applying the Kumar et al. algorithm. C. Limitations of Prior Art Prior methods have three fundamental limitations. First, they follow the Union then Minimize framework whih means they reate large automata and only minimize them at the end. This also means they must employ the expensive NFA to DFA suset onstrution. Seond, prior methods uild the orresponding minimum state DFA efore onstruting the final D 2 FA. This is very ostly in oth spae and time. The D 2 FA is typially 50 to 100 times smaller than the DFA, so even if the D 2 FA would fit in availale memory, the intermediate DFA might e too large, making it impratial to uild the D 2 FA. This is exaerated in the ase of the Kumar et al. algorithm whih needs the SRG whih ranges from aout the size of the DFA itself to over 50 times the size of the DFA. The resulting spae and time required to uild the DFA and SRG impose serious limits on the D 2 FA that an e pratially onstruted. We do oserve that the method proposed in [15] does not need to reate the SRG. Furthermore, as the authors have noted, there is a way to go from the NFA diretly to the D 2 FA, ut implementing suh an approah is still very ostly in time as many transition tales need to e repeatedly rereated in order to realize these spae savings. Third, none of the previous methods provide effiient algorithms for updating the D 2 FA when a new RE is added to R. D. Our Approah To address these limitations, we propose a Minimize then Union framework. Speifially, we first minimize the small automata orresponding to eah RE fromrand then union the minimized automata together. A key property of our method is that our union algorithm automatially produes a minimum state D 2 FA for the regular expressions involved without expliit state minimization. Likewise, we hoose deferment states effiiently while performing the union operation using deferment information from the input D 2 FAs. Together, these optimizations lead to a vastly more effiient D 2 FA onstrution algorithm in oth time and spae. In more detail, given R, we first uild a DFA and D 2 FA for eah individual RE in R. The heart of our tehnique is

3 3 Fig. 1. from 1,3 {a,} 0 a {a,,} from 2,5,8,11 {,} 1 3 from from 4,6,7,9,10,12 4,6,10 {,} 4 a 2 a from 5,8, / /1,2 8 11/ {a,} 0 a {,} a 2 {a,} (a) () () (a) DFA for {.*a.*,.*.*}. () Corresponding SRG. Edges of weight 1 not shown. Unlaeled edges have weight 255. () The D 2 FA. the D 2 FA merge algorithm that performs the union. It merges two smaller D 2 FAs into one larger D 2 FA suh that the merged D 2 FA is equivalent to the union of REs that the D 2 FAs eing merged were equivalent to. Starting from the the initial D 2 FAs for eah RE, using this D 2 FA merge suroutine, we merge two D 2 FAs at a time until we are left with just one final D 2 FA. The initial D 2 FAs are eah equivalent to their respetive REs, so the final D 2 FA will e equivalent to the union of all the REs in R. Figures 2(a) and 2() show the initial D 2 FAs for the RE set {.*a.*,.*.*}. The resulting D 2 FA from merging these two D 2 FAs using the D 2 FA merge algorithm is shown in Figure 2(). The D 2 FA produed y our merge algorithm an e larger than the minimal D 2 FA produed y the Kumar et al. algorithm. This is eause the Kumar et al. algorithm does a gloal optimization over the whole DFA (using the SRG), whereas our merge algorithm effiiently omputes state deferment in the merged D 2 FA ased on state deferment in the two input D 2 FAs. In most ases, the D 2 FA produed y our approah is suffiiently small to e deployed. However, in situations where more ompression is needed, we offer an effiient final ompression algorithm that produes a D 2 FA very similar in size to that produed y the Kumar et al. algorithm. This final ompression algorithm uses an SRG; we improve effiieny y using the deferment already omputed in the merged D 2 FA to greatly redue the size of this SRG and thus signifiantly redue the time and memory required to do this ompression. a) Advantages of our algorithm: One of the main advantages of our algorithm is a dramati inrease in time and spae effiieny. These effiieny gains are partly due to our use of the Minimize then Union framework instead of the Union then Minimize framework. More speifially, our improved effiieny omes aout from the following four fators. First, other than for the initial DFAs that orrespond to individual REs in R, we uild D 2 FA ypassing DFAs. Those initial DFAs are very small (typially < 50 states), so the memory and time required to uild the initial DFAs and D 2 FAs is negligile. The D 2 FA merge algorithm diretly merges the two input D 2 FAs to get the output D 2 FA without reating the DFA first. Seond, other than for the initial DFAs, we never have to perform the NFA to DFA suset onstrution. Third, other than for the initial DFAs, we never have to perform DFA state minimization. Fourth, when setting deferment states in the D 2 FA merge algorithm, we use deferment information from the two input D 2 FA. This typially involves performing /1 12/1,2 only a onstant numer of omparisons per state rather than a linear in the numer of states omparison per state as is required y previous tehniques. All told, our algorithm has a pratial time omplexity of O(n Σ ) where n is the numer of states in the final D 2 FA and Σ is the size of the input alphaet. In ontrast, Kumar et al. s algorithm [7] has a time omplexity of O(n 2 (log(n)+ Σ )) and Behi and Crowley s algorithm [15] has a time omplexity of O(n 2 Σ ) just for setting the deferment state for eah state and ignoring the ost of the NFA suset onstrution and DFA state minimization. See Setion V-D for a more detailed omplexity analysis. These effiieny advantages allow us to uild muh larger D 2 FAs than are possile with previous methods. For the syntheti RE set that we onsider in Setion VI, given a maximum working memory size of 1GB, we an uild a D 2 FA with 80,216,064 states with our D 2 FA merge algorithm whereas the Kumar et al. algorithm an only uild a D 2 FA with 397,312 states. Also from Setion VI, our algorithm is typially 35 to 250 times faster than previous algorithms on our RE sets. Besides eing muh more effiient in onstruting D 2 FA from srath, our algorithm is very well suited for frequent RE updates. When an RE needs to e added to the urrent set, we just need to merge the D 2 FA for the RE to the urrent D 2 FA using our merge routine whih is a very fast operation. ) Tehnial Challenges: For our approah to work, the main hallenge is to figure out how to effiiently union two minimum state D 2 FAs D 1 and D 2 so that the resulting D 2 FA D 3 is also a minimum state D 2 FA. There are two aspets to this hallenge. First, we need to make sure that D 2 FA D 3 has the minimum numer of states. More speifially, suppose D 1 and D 2 are equivalent to RE sets R 1 and R 2, respetively. We need to ensure that D 3 has the minimum numer of states of any D 2 FA equivalent to R 1 R 2. We use an existing union ross produt onstrution for this and prove that it results in a minimum state D 2 FA for our purpose. We emphasize that this is not true in general ut holds for appliations where D 3 must identify whih REs from R 1 R 2 math a given input string. Many seurity appliations meet this riteria. Our seond hallenge is uilding the D 2 FA D 3 without uilding the entire DFA equivalent to D 3 while ensuring that D 3 ahieves signifiant transition ompression; that is, the numer of atual edges stored in D 2 FA D 3 must e small. More onretely, as eah state in D 3 is reated, we need to immediately set a deferred state for it; otherwise, we would e 11/2

4 4 a a /1 (a) /2 () {a,} {a,} 0,0 0 0,1 2 0,2 5 0,3 8 0,4 11/2 () (d) Fig. 2. (a) D 1 : D 2 FA for.*a.*. () D 2 : D 2 FA for.*.*. () D 3 : merged D 2 FA. (d) Illustration of setting deferment for some states in D 3. storing the entire DFA. Furthermore, we need to hoose a good deferment state that eliminates as many edges as possile. We address this hallenge y effiiently hoosing a good deferment state for D 3 y using the deferment information from D 1 and D 2. Typially, the algorithm only needs to ompare a handful of andidate deferment states. ) Key Contriutions: In summary, we make the following ontriutions: (1) We propose a novel Minimize then Union framework for effiiently onstruing D 2 FA for network seurity RE sets that we elieve will generalize to other RE mathing automata. Note, Smith et al. used the Minimize then Union Framework when onstruting XFA [11], [12], though they did not prove any properties aout their union algorithms. (2) To implement this framework, we propose a very effiient D 2 FA merge algorithm for performing the union of two D 2 FAs. (3) When maximum ompression is needed, we propose an effiient final ompression step to produe a nearly minimal D 2 FA. (4) To prove the orretness of our D 2 FA merge algorithm, we prove a fundamental property aout the standard union ross produt onstrution and minimum state DFAs when applied to network seurity RE sets that an e applied to other RE mathing automata. We implemented our algorithms and onduted experiments on real-world and syntheti RE sets. Our experiments indiate that: (a) Our algorithm generates D 2 FA with a fration of the memory required y existing algorithms making it feasile to uild D 2 FA with many more states. For the real world RE sets we onsider in the experimental setion, our algorithm requires an average of 1500 times less memory than the algorithm proposed in [7] and 30 times less memory than the algorithm proposed in [15]. () Our algorithm runs muh faster then existing algorithms. For the real world RE sets we onsider in the experimental setion, our algorithm runs an average of 154 times faster than the algorithm proposed in [7] and 19 times faster than the algorithm proposed in [15]. () Even with the huge spae and time effiieny gains, our algorithm generates D 2 FA only slightly larger than existing algorithms in the worst ase. If the resulting D 2 FA is too large, our effiient final ompression algorithm produes a nearly minimal D 2 FA. a a {,} 1,0 1 1,1 4 2,0 3 2,2 7 3,1 6 3,3 10 4,2 9/1 4,4 12/1, Deferment for 5= 0, Deferment for 7= 2, Deferment for 9= 4, Deferment for 12= 4,4 II. RELATED WORK Initially network intrusion detetion and prevention systems used string patterns to speify attak signatures [16] [22]. Sommer and Paxson [4] first proposed using REs instead of strings to speify attak signatures. Today network intrusion detetion and prevention systems mostly use REs for attak signatures. RE mathing solutions are typially software-ased or hardware-ased (FPGA or ASIC). Software-ased approahes are heap and deployale on general purpose proessors, ut their throughput may not e high. To ahieve higher throughput, software solutions an e deployed on ustomized ASIC hips at the ost of low versatility and high deployment ost. To ahieve deterministi throughput, software-ased solutions must use DFAs, whih fae a spae explosion prolem. Speifially, there an e state explosion where the numer of states inreases exponentially in the numer of REs, and the numer of transitions per state is extremely high. To address the spae explosion prolem, transition ompression and state minimization software-ased solutions have een developed. Transition ompression shemes that minimize the numer of transitions per state have mostly used one of two tehniques. One is alphaet re-enoding, whih exploits redundany within a state, [15], [23] [25]. The seond is default transitions or deferment states whih exploits redundany among states [7], [8], [15], [26]. Kumar et al. [7] originally proposed the use of default transitions. Behi and Crowley [15] proposed a more effiient way of using default transitions. Our work falls into the ategory of transition ompression via default transitions. Our algorithms are muh more effiient than those of [7], [15] and thus an e applied to muh larger RE sets. For example, if we are limited to 1GB of memory to work with, we show that Kumar et al. s original algorithm an only uild a D 2 FA with less than 400,000 states whereas our algorithm an uild a D 2 FA with over 80,000,000 states. Two asi approahes have een proposed for state minimization. One is to partition the given RE set and uild a DFA for eah partition [27]. When inspeting paket payload, eah input symol needs to e sanned against eah partition s DFA. Our work is orthogonal to this tehnique and an e used

5 5 in omination with this tehnique. The seond approah is to modify the automata struture and/or use extra memory to rememer history and thus avoid state dupliation [10] [14]. We elieve our merge tehnique an e adopted to work with some of these approahes. For example, Smith et al. also use the Minimize then Union framework when onstruting XFA [11], [12]. One potential drawak with XFA is that there is no fully automated proedure to onstrut XFAs from a set of regular expressions. Paraphrasing Yang, Karim, Ganapathy, and Smith [28], onstruting an XFA from a set of REs requires manual analysis of the REs to identify and eliminate amiguity. FPGA-ased solutions typially exploit the parallel proessing apailities of FPGAs to implement a Nondeterministi Finite Automata (NFA) [13], [24], [29] [33] or to implement multiple parallel DFAs [34]. TCAM ased solutions have een proposed for string mathing in [20] [22], [35] and for REs in [9]. Our work an potentially e applied to these solutions as well. Reently and independently, Liu et al. proposed to onstrut DFA y hierarhial merging [36]. That is, they essentially propose the Minimize then Union framework for DFA onstrution. They onsider merging multiple DFAs at a time rather than just two. However, they do not onsider D 2 FA, and they do not prove any properties aout their merge algorithm inluding that it results in minimum state DFAs. III. PATTERN MATCHING DFAS A. Pattern Mathing DFA Definition In a standard DFA, defined as a 5-tuple (Q,Σ,δ,q 0,A), eah aepting state is equivalent to any other aepting state. However, in many pattern mathing appliations where we are given a set of REs R, we must keep trak of whih REs have een mathed. For example, eah RE may orrespond to a unique seurity threat that requires its own proessing routine. This leads us to define Pattern Mathing Deterministi Finite State Automata (PMDFA). The key differene etween a PMDFA and a DFA is that for eah state q in a PMDFA, we annot simply mark it as aepting or rejeting; instead, we must reord whih REs from R are mathed when we reah q. More formally, given as input a set of REs R, a PMDFA is a 5-tuple (Q,Σ,δ,q 0,M) where the last term M is now defined as M: Q 2 R. B. Minimum State PMDFA onstrution Given a set of REs R, we an uild the orresponding minimum state PMDFA using the standard Union then Minimize framework: first uild an NFA for the RE that orresponds to an OR of all the REs r R, then onvert the NFA to a DFA, and finally minimize the DFA treating aepting states as equivalent if and only if they orrespond to the same set of regular expressions. This method an e very slow, mainly due to the NFA to DFA onversion, whih often results in an exponential growth in the numer of states. Instead, we propose a more effiient Minimize then Union framwork. Let R 1 and R 2 denote any two disjoint susets of R, and let D 1 and D 2 e their orresponding minimum state PMDFAs. We use the standard union ross produt onstrution to onstrut a minimum state PMDFA D 3 that orresponds to R 3 = R 1 R 2. Speifially, suppose we are given the two PMDFAs D 1 = (Q 1,Σ,δ 1,q 01,M 1 ) and D 2 = (Q 2,Σ,δ 2,q 02,M 2 ). The union ross produt PMDFA of D 1 and D 2, denoted as UCP(D 1,D 2 ), is given y D 3 = UCP(D 1,D 2 ) = (Q 3,Σ,δ 3,q 03,M 3 ) where Q 3 = Q 1 Q 2, δ 3 ( q i,q j,x) = δ 1 (q i,x),δ 2 (q j,x), q 03 = q 01,q 02, and M 3 ( q i,q j ) = M 1 (q i ) M 2 (q j ). Eah state in D 3 orresponds to a pair of states, one from D 1 and one from D 2. For notational larity, we use and to enlose an ordered pair of states. Transition funtion δ 3 just simulates oth δ 1 and δ 2 in parallel. Many states in Q 3 might not e reahale from the start state q 03. Thus, while onstruting D 3, we only reate states that are reahale from q 03. We now argue that this onstrution is orret. This is a standard onstrution, so the fat that D 3 is a PMDFA for R 3 = R 1 R 2 is straightforward and overed in standard automata theory textooks (e.g. [5]). We now show that D 3 is also a minimum state PMDFA for R 3 assuming R 1 R 2 =, a result that does not follow for standard DFAs. Theorem III.1. Given two RE sets, R 1 and R 2, and equivalent minimum state PMDFAs, D 1 and D 2, the union ross produt DFA D 3 = UCP(D 1,D 2 ), with only reahale states onstruted, is the minimum state PMDFA equivalent to R 3 = R 1 R 2 if R 1 R 2 =. Proof: First sine only reahale states are onstruted, D 3 annot e trivially redued. Now assume D 3 is not minimum. That would mean there are two states in D 3, say p 1,p 2 and q 1,q 2, that are indistinguishale. This implies that x Σ, M 3 (δ 3 ( p 1,p 2,x)) = M 3 (δ 3 ( q 1,q 2,x)). Working on oth sides of this equality, we get x Σ, M 3 (δ 3 ( p 1,p 2,x)) = M 3 ( δ 1 (p 1,x),δ 2 (p 2,x) ) as well as x Σ, = M 1 (δ 1 (p 1,x)) M 2 (δ 2 (p 2,x)) M 3 (δ 3 ( q 1,q 2,x)) = M 3 ( δ 1 (q 1,x),δ 2 (q 2,x) ) This implies that = M 1 (δ 1 (q 1,x)) M 2 (δ 2 (q 2,x)) x Σ M 1 (δ 1 (p 1,x)) M 2 (δ 2 (p 2,x)) = M 1 (δ 1 (q 1,x)) M 2 (δ 2 (q 2,x)). Now sine R 1 R 2 =, this gives us x Σ, M 1 (δ 1 (p 1,x)) = M 1 (δ 1 (q 1,x)) and x Σ, M 2 (δ 1 (p 2,x)) = M 2 (δ 1 (q 2,x)) This implies that p 1 and q 1 are indistinguishale in D 1 and p 2 and q 2 are indistinguishale in D 2, implying that oth D 1 and D 2 are not minimum state PMDFAs, whih is a ontradition and the result follows. Our effiient onstrution algorithm works as follows. First, for eah RE r R, we uild an equivalent minimum state PMDFA D for r using the standard method, resulting in a set

6 6 of PMDFAs D. Then we merge two PMDFAs from D at a time using the aove UCP onstrution until there is just one PMDFA left in D. The merging is done in a greedy manner: in eah step, the two PMDFAs with the fewest states are merged together. Note the ondition R 1 R 2 = is always satisfied in all the merges. In our experiments, our Minimize then Union tehnique runs exponentially faster than the standard Union then Minimize tehnique eause we only apply the NFA to DFA step to the NFAs that orrespond to eah individual regular expression rather than the omposite regular expression. This makes a signifiant differene even when we have a relatively small numer of regular expressions. For example, for our C7 RE set whih ontains 7 REs, the standard tehnique requires seonds to uild the PMDFA, ut our tehnique uilds the PMDFA in only 0.66 seonds. For the remainder of this paper, we use DFA to stand for minimum state PMDFA. IV. D 2 FA CONSTRUCTION In this setion, we first formally define what a D 2 FA is and then desrie how we an extend the Minimize then Union tehnique to D 2 FA ypassing DFA onstrution. A. D 2 FA Definition Let D = (Q,Σ,δ,q 0,M) e a DFA. A orresponding D 2 FA D is defined as a 6-tuple (Q,Σ,ρ,q 0,M,F). Together, funtion F: Q Q and partial funtion ρ: Q Σ Q are equivalent to DFA transition funtion δ. Speifially, F defines a unique deferred state for eah state in Q, and ρ is a partially defined transition funtion. We use dom(ρ) to denote the domain of ρ, i.e. the values for whih ρ is defined. The key property of a D 2 FA D that orresponds to DFA D is that q, Q Σ, q, dom(ρ) (F(q) = q δ(q,) δ(f(q),)); that is for eah state, ρ only has those transitions that are different from that of its deferred state in the underlying DFA. When defined, ρ(q,) = δ(q,). States that defer to themselves must have all their transitions defined. We only onsider D 2 FA that orrespond to minimum state DFA, though the definition applies to all DFA. The funtion F defines a direted graph on the states of Q. A D 2 FA is well defined if and only if there are no yles of length > 1 in this direted graph whih we all a deferment forest. We use p q to denote F(p) = q, i.e. p diretly defers to q. We use p q to denote that there is a path from p to q in the deferment forest defined y F. We use p q to denote the numer of transitions in ommon etween states p and q; i.e. p q = { Σ δ(p,) = δ(q,)}. The total transition funtion for a D 2 FA is defined as { δ ρ(u,) if u, dom(ρ) (u,) = δ (F(u),) else It is easy to see that δ is well defined and equal to δ if the D 2 FA is well defined. B. D 2 FA Merge Algorithm The UCP onstrution merges two DFAs together. We extend the UCP onstrution to merge two D 2 FAs together as follows. During the UCP onstrution, as eah new state u is reated, we define F(u) at that time. We then define ρ to only inlude transitions for u that differ from F(u). To help explain our algorithm, Figure 2 shows an example exeution of the D 2 FA merge algorithm. Figures 2(a) and 2() show the D 2 FA for the REs.*a.* and.*.*, respetively. Figure 2() shows the merged D 2 FA for the D 2 FAs in figures 2(a) and 2(). We use the following onventions when depiting a D 2 FA. The dashed lines orrespond to the deferred state for a given state. For eah state in the merged D 2 FA, the pair of numers aove the line refer to the states in the original D 2 FAs that orrespond to the state in the merged D 2 FA. The numer elow the line is the state in the merged D 2 FA. The numer(s) after the / in aepting states give the id(s) of the pattern(s) mathed. Figure 2(d) shows how the deferred state is set for a few states in the merged D 2 FAs D 3. We explain the notation in this figure as we give our algorithm desription. For eah state u D 3, we set the deferred state F(u) as follows. While merging D 2 FAs D 1 and D 2, let state u = p 0,q 0 e the new state urrently eing added to the merged D 2 FA D 3. Let p 0 p 1 p l e the maximal deferment hain DC 1 (i.e. p l defers to itself) in D 1 starting at p 0, and q 0 q 1 q m e the maximal deferment hain DC 2 in D 2 starting atq 0. For example, in Figure 2 (d), we see the maximal deferment hains for u = 5 = 0,2, u = 7 = 2,2, u = 9 = 4,2, and u = 12 = 4,4. For u = 9 = 4,2, the top row is the deferment hain of state 4 in D 1 and the ottom row is the deferment hain of state 2 in D 2. We will hoose some state p i,q j where 0 i l and 0 j m to e F(u). In Figure 2(d), we represent these andidate F(u) pairs with edges etween the nodes of the deferment hains. For eah andidate pair, the numer on the top is the orresponding state numer in D 3 and the numer on the ottom is the numer of ommon transitions in D 3 etween that pair and state u. For example, for u = 9 = 4, 2, the two andidate pairs represented are state 7 ( 2, 2 ) whih shares 256 transitions in ommon with state 9 and state 4 ( 1,1 ) whih shares 255 transitions in ommon with state 9. Note that a andidate pair is only onsidered if it is reahale in D 3. In Figure 2(d) with u = 9 = 4,2, three of the andidate pairs orresponding to 4,1, 2,1, and 1,2 are not reahale, so no edge is inluded for these andidate pairs. Ideally, we want i and j to e as small as possile though not oth 0. For example, our est hoies are typially p 0,q 1 or p 1,q 0. In the first ase, p 0 p 1 = p 0,q 0 p 1,q 0, and we already have p 0 p 1 in D 1. In the seond ase, q 0 q 1 = p 0,q 0 p 0,q 1, and we already have q 0 q 1 in D 2. In Figure 2 (d), we set F(u) to e p 0,q 1 for u = 5 = 0,2 and u = 12 = 4,4, and we use p 1,q 0 for u = 9 = 4,2. However, it is possile that oth states are not reahale from the start state in D 3. This leads us to onsider other possile p i,q j. For example, in Figure 2 (d), oth 2,1 and 1,2 are not reahale in D 3, so we use reahale state 1,1 as F(u) for u = 7 = 2,2. We onsider a few different algorithms for hoosing p i,q j. The first algorithm whih we all the first math method is to find a pair of states (p i, q j ) for whih p i,q j Q 3 and i+j is minimum. Stated another way, we find the minimum z 1 suh that the set of states Z = { p i,q z i (max(0,z m) i min(l,z)) ( p i,q z i Q 3 )}. From the set of states Z, we hoose the state that has the most transitions in ommon

7 7 with p 0,q 0 reaking ties aritrarily. If Z is empty for all z > 1, then we just pik p 0,q 0, i.e. the deferment pointer is not set (or the state defers to itself). The idea ehind the first math method is that p 0,q 0 p i,q j dereases as i+j inreases. In Figure 2(d), all the seleted F(u) orrespond to the first math method. A seond more omplete algorithm for setting F(u) is the est math method where we always onsider all (l + 1) (m + 1) 1 pairs and pik the pair that is in Q 3 and has the most transitions in ommon with p 0,q 0. The idea ehind the est math method is that it is not always true that p 0,q 0 p x,q y p 0,q 0 p x+i,q y+j for i+j > 0. For instane we an have p 0 p 2 < p 0 p 3, whih would mean p 0,q 0 p 2,q 0 < p 0,q 0 p 3,q 0. In suh ases, the first math method will not find the pair along the deferment hains with the most transitions in ommon with p 0,q 0. In Figure 2(d), all the seleted F(u) also orrespond to the est math method. It is diffiult to reate a small example where first math and est math differ. When adding the new state u to D 3, it is possile that some state pairs along the deferment hains that were not in Q 3 while finding the deferred state for u will later on e added to Q 3. This means that after all the states have een added to Q 3, the deferment for u an potentially e improved. Thus, after all the states have een added, for eah state we again find a deferred state. If the new deferred state is etter than the old one, we reset the deferment to the new deferred state. Algorithm 1 shows the pseudoode for the D 2 FA merge algorithm with the first math method for hoosing a deferred state. Note that we use u and u 1,u 2 interhangealy to indiate a state in the merged D 2 FA D 3 where u is a state in Q 3, and u 1 and u 2 are the states in Q 1 and Q 2, respetively, that state u orresponds to. C. Original D 2 FA onstrution for one RE Before we an merge D 2 FAs, we first must onstrut a D 2 FA for eah RE. One option, whih we used in the NDSS preliminary version of this paper, is the D 2 FA onstrution algorithm proposed in [9] whih is ased on the original D 2 FA onstrution algorithm proposed in [7]. This is an effetive algorithm whih we now desrie in more detail. The first step is to uild the Spae Redution Graph (SRG): a omplete graph where the verties represent DFA states and the weight of eah SRG edge is the numer of ommon transitions etween its end points in the DFA. Meiners et al. note that for real world RE sets, the distriution of edge weights in the SRG is imodal, with edge weights typially either very small (< 10) or very large (> 180). They hose to omit low (< 10) weight edges from the SRG whih then produed a forest with many distint onneted omponents. They then onstrut a maximum spanning forest of the SRG using Kruskal s algorithm. The next step is to hoose a root state for eah onneted omponent of the SRG. For this, Meiners et al. hoose selflooping states to e root states. A state is a self-looping state if it has more than half (i.e. 128) of its transitions looping ak to itself. Eah omponent of the SRG has at most one self-looping state. For omponents that do not have a selflooping state, they hoose one of the states in the enter of Algorithm 1: D2FAMerge(D 1,D 2 ) Input: A pair of D 2 FAs, D 1 = (Q 1,Σ,ρ 1,q 01,M 1,F 1) and D 2 = (Q 2,Σ,ρ 2,q 02,M 2,F 2), orresponding to RE sets, say R 1 and R 2, with R 1 R 2 =. Output: A D 2 FA orresponding to the RE set R 1 R 2 1 Initialize D 3 to an empty D 2 FA; 2 Initialize queue as an empty queue; 3 queue.push ( q 01,q 02 ); 4 while queue not empty do 5 u = u 1, u 2 := queue.pop(); 6 Q 3 := Q 3 {u}; 7 foreah Σ do 8 nxt := δ 1 (u1, ),δ 2 (u2, ) ; 9 if nxt / Q 3 nxt / queue then queue.push (nxt); 10 Add (u, ) nxt transition to ρ 3; 11 M 3(u) := M 1(u 1) M 2(u 2); 12 F 3(u) := FindDefState(u); 13 Remove transitions for u from ρ 3 that are in ommon with F 3(u); 14 foreah u Q 3 do 15 newdptr := FindDefState(u); 16 if (newdptr F 3(u)) (newdptr u > F 3(u) u) then 17 F 3(u) := newdptr; 18 Reset all transitions for u in ρ 3 and then remove ones that are in ommon with F 3(u); 19 return D 3; 20 FindDefState ( v 1,v 2 ) 21 Let p 0 = v 1,p 1,...,p l e the list of states on the deferment hain from v 1 to the root in D 1; 22 Let q 0 = v 2,q 1,...,q m e the list of states on the deferment hain from v 2 to the root in D 2; 23 for z = 1 to (l + m) do 24 S := { p i,q z i (max(0,z m) i min(l,z)) ( p i,q z i Q 3)}; 25 if S then return argmax v S ( v 1,v 2 v); 26 return v 1,v 2 ; the spanning tree as the root state. After hoosing the root for eah tree, all the edges in the spanning tree are direted towards the root, giving the deferment pointer for eah state. One sutle point of this algorithm is that there are many ases where multiple edges an e added to the spanning tree. Speifially, Kruskal s algorithm always hooses the edge with the maximum weight from the remaining edges. Sine there are only 256 possile edge weights, there often are multiple edges with the same maximum weight. Meiners et al. use the following tie reaking order among edges having the urrent maximum weight. 1) Edges that have a self-looping state as one of their end points are given the highest priority. 2) Next, priority is given to edges with higher sum of degrees (in the urrent spanning tree) of their end verties. D. Improved D 2 FA onstrution for one RE We now offer an improved algorithm for onstruting a D 2 FA for one RE. This algorithm is similar to that of Meiners et al. s algorithm [9]. The differene is we modify and extend the tie-reaking strategy as follows. For eah state u, we store a value, deg (u), whih is initially set to 0. During Kruskal s algorithm, when an edge e = (u,v) is added to the urrent spanning tree, deg (u) is inremented y 2 if level(u) level(v); otherwise it is inremented y 1. Reall that level(u) is the length of the shortest string that takes the DFA from the start state to state u. We similarly

8 8 update deg (v). Then we use the following tie reaking order among edges having the urrent maximum weight. 1) Edges that have a self-looping state as one of their end points are given the highest priority. 2) Next, priority is given to edges with higher sum of deg of their end verties. 3) Next, priority is given to edges with higher differene etween the levels of their end verties. The sum of degrees of end verties is used for tie reaking in order to prioritize states that are already highly onneted. However, we also want to prioritize onneting to states at lower levels, so we use deg instead of just the degree. Using the differene etween levels of end points for tie reaking also prioritizes states at a lower level. This helps redue the deferment depth and the D 2 FA size for RE sets whose D 2 FAs have a higher average deferment depth. We oserve in our experiments setion that the improved algorithm does outperform the original algorithm. E. D 2 FA onstrution for RE set R We now have methods for onstruting a D 2 FA given one RE and merging two D 2 FAs into one D 2 FA. We omine these methods in the natural way to uild one D 2 FA for a set of REs. That is, we first uild a D 2 FA for eah RE in R. We then merge the D 2 FAs together using a alaned inary tree struture to minimize the worst-ase numer of merges that any RE experienes. We do use two different variations of our D2FAMerge algorithm. For all merges exept the final merge, we use the first math method for setting F(u). When doing the final merge to get the final D 2 FA, we use the est math method for setting F(u). It turns out that using the first math method results in a etter deferment forest struture in the D 2 FA, whih helps when the D 2 FA is further merged with other D 2 FAs. The loal optimization ahieved y using the est math method only helps when used in the final merge. F. Optional Final Compression Algorithm When there is no ound on the deferment depth (see Setion V-B), the original D 2 FA algorithm proposed in [7] results in a D 2 FA with smallest possile size eause it runs Kruskal s algorithm on a large SRG. Our D 2 FA merge algorithm results in a slightly larger D 2 FA eause it uses a greedy approah to determine deferment. We an further redue the size of the D 2 FA produed y our algorithm y running the following ompression algorithm on the D 2 FA produed y the D 2 FA merge algorithm. We onstrut an SRG and perform a maximum weight spanning tree onstrution on the SRG, ut we only add edges to the SRG that have the potential to redue the size of the D 2 FA. More speifially, let u and v e any two states in the urrent D 2 FA. We only add the edge e = (u,v) in the SRG if its weight w(e) is min(u F(u),v F(v)). Here, F(u) is the deferred state of u in the urrent D 2 FA. As a result, very few edges are added to the SRG, so we only need to run Kruskal s algorithm on a small SRG. This saves oth spae and time ompared to previous D 2 FA onstrution methods. However, this ompression step does require more time and spae than the D 2 FA merge algorithm eause it does onstrut an SRG and then runs Kruskal s algorithm on the SRG. V. D 2 FA MERGE ALGORITHM PROPERTIES A. Proof of Corretness The D 2 FA merge algorithm exatly follows the UCP onstrution to reate the states. So the orretness of the underlying DFA follows from the the orretness of the UCP onstrution. Theorem V.1 shows that the merged D 2 FA is also well defined (no yles in deferment forest). Lemma V.1. In the D 2 FA D 3 = D2FAMerge(D 1,D 2 ), u 1,u 2 v 1,v 2 u 1 v 1 u 2 v 2. Proof: If u 1,u 2 = v 1,v 2 then the lemma is trivially true. Otherwise, let u 1,u 2 w 1,w 2 v 1,v 2 e the deferment hain in D 3. When seleting the deferred state for u 1,u 2, D2FA Merge always hoose a state that orresponds to a pair of states along deferment hains for u 1 and u 2 in D 1 and D 2, respetively. Therefore, we have that u 1,u 2 w 1,w 2 u 1 w 1 u 2 w 2. By indution on the length of the deferment hain and the fat that the relation is transitive, we get our result. Theorem V.1. If D 2 FAs D 1 and D 2 are well defined, then the D 2 FA D 3 = D2FAMerge(D 1,D 2 ) is also well defined. Proof: Sine D 1 and D 2 are well defined, there are no yles in their deferment forests. Now assume that D 3 is not well defined, i.e. there is a yle in its deferment forest. Let u 1,u 2 and v 1,v 2 e two distint states on the yle. Then, we have that u 1,u 2 v 1,v 2 v 1,v 2 u 1,u 2 Using Lemma V.1 we get (u 1 v 1 u 2 v 2 ) (v 1 u 1 v 2 u 2 ) i.e. (u 1 v 1 v 1 u 1 ) (u 2 v 2 v 2 u 2 ) Sine u 1,u 2 v 1,v 2, we have u 1 v 1 u 2 v 2 whih implies that at least one of D 1 or D 2 has a yle in their deferment forest whih is a ontradition. B. Limiting Deferment Depth Sine no input is onsumed while traversing a deferred transition, in the worst ase, the numer of lookups needed to proess one input harater is given y the depth of the deferment forest. As previously proposed, we an guarantee a worst ase performane y limiting the depth of the deferment forest. For a state u 1 of a D 2 FA D 1, the deferment depth of u 1, denoted asψ(u 1 ), is the length of the maximal deferment hain in D 1 from u 1 to the root. Ψ(D 1 ) = max v Q1 ψ(v) denotes the deferment depth of D 1 (i.e. the depth of the deferment forest in D 1 ). Lemma V.2. In the D 2 FA D 3 = D2FAMerge(D 1,D 2 ), u 1,u 2 Q 3, ψ( u 1,u 2 ) ψ(u 1 )+ψ(u 2 ). Proof: Let ψ( u 1,u 2 ) = d. If ψ( u 1,u 2 ) = 0, then u 1,u 2 is a root and the lemma is trivially true. So, we onsider d 1 and assume the lemma is true for all states

9 9 withψ < d. Let u 1,u 2 w 1,w 2 v 1,v 2 e the deferment hain in D 3. Using the indutive hypothesis, we have ψ( w 1,w 2 ) ψ(w 1 )+ψ(w 2 ) Given u 1,u 2 w 1,w 2, we assume without loss of generality that u 1 w 1. Using Lemma V.1 we get that u 1 w 1. Therefore ψ(w 1 ) ψ(u 1 ) 1. Comining the aove, we get ψ( u 1,u 2 ) = ψ( w 1,w 2 )+1 ψ(w 1 )+ψ(w 2 )+1 (ψ(u 1 ) 1)+ψ(u 2 )+1 ψ(u 1 )+ψ(u 2 ). Lemma V.2 diretly gives us the following Theorem. Theorem V.2. If D 3 = D2FAMerge(D 1,D 2 ), then Ψ(D 3 ) Ψ(D 1 )+Ψ(D 2 ). For an RE set R, if the initial D 2 FAs have Ψ = d, in the worst ase, the final merged D 2 FA orresponding to R an have Ψ = d R. Although Theorem V.2 gives the value of Ψ in the worst ase, in pratial ases, Ψ(D 3 ) is very lose to max(ψ(d 1 ),Ψ(D 2 )). Thus the deferment depth of the final merged D 2 FA is usually not muh higher than d. Let Ω denote the desired upper ound on Ψ. To guarantee Ψ(D 3 ) Ω, we modify the FindDefState suroutine in Algorithm 1 as follows: When seleting andidate pairs for the deferred state, we only onsider states with ψ < Ω. Speifially, we replae line 24 with the following S := { p i,q z i (max(0,z m) i min(l,z)) p i,q z i Q 3) (ψ( p i,q z i ) < Ω)} When we do the seond pass (lines 14-19), we may inrease the deferment depth of nodes that defer to nodes that we readjust. We reord the affeted nodes and then do a third pass to reset their deferment states so that the maximum depth ound is satisfied. In pratie, this happens very rarely. When onstruting a D 2 FA with a given ound Ω, we first uild D 2 FAs without this ound. We only apply the ound Ω when performing the final merge of two D 2 FAs to reate the final D 2 FA. C. Deferment to a Lower Level In [15], the authors propose a tehnique to guarantee an amortized ost of 2 lookups per input harater without limiting the depth of the deferment tree. They ahieve this y having states only defer to lower level states where the level of any state u in a DFA (or D 2 FA), denoted level(u), is defined as the length of the shortest string that ends in that state (from the start state). More formally, they ensure that for all states u, level(u) > level(f(u)) if u F(u). We all this property the ak-pointer property. If the ak-pointer property holds, then every deferred transition taken dereases the level of the urrent state y at least 1. Sine a regular transition on an input harater an only inrease the level of the urrent state y at most 1, there have to e fewer deferred transitions taken on the entire input string than regular transitions. This gives an amortized ost of at most 2 transitions taken per input harater. In order to guarantee the D 2 FA D 3 has the akpointer property, we perform a similar modifiation to the FindDefState suroutine in Algorithm 1 as we performed when we wanted to limit the maximum deferment depth. When seleting andidate pairs for the deferred state, we only onsider states with a lower level. Speifially, we replae line 24 with the following: S := { p i,q z i (max(0,z m) i min(l,z)) ( p i,q z i Q 3) (level( v 1,v 2 ) > level( p i,q z i ))} For states for whih no andidate pairs are found, we just searh through all states in Q 3 that are at a lower level for the deferred state. In pratie, this searh through all the states needs to e done for very few states eause if D 2 FAs D 1 and D 2 have the ak-pointer property, then almost all the states in D 2 FAs D 3 have the ak-pointer property. As with limiting maximum deferment depth, we only apply this restrition when performing the final merge of two D 2 FAs to reate the final D 2 FA. D. Algorithmi Complexity The time omplexity of the original D 2 FA algorithm proposed in [7] is O(n 2 (log(n) + Σ )). The SRG has O(n 2 ) edges, and O( Σ ) time is required to add eah edge to the SRG and O(log(n)) time is required to proess eah edge in the SRG during the maximum spanning tree routine. The time omplexity of the D 2 FA algorithm proposed in [15] is O(n 2 Σ ). Eah state is ompared with O(n) other states, and eah omparison requires O( Σ ) time. The time omplexity of our new D2FAMerge algorithm to merge two D 2 FAs is O(nΨ 1 Ψ 2 Σ ), where n is the numer of states in the merged D 2 FA, and Ψ 1 and Ψ 2 are the maximum deferment depths of the two input D 2 FAs. When setting the deferment for any state u = u 1,u 2, in the worst ase the algorithm ompares u 1,u 2 with all the pairs along the deferment hains of u 1 and u 2, whih are at most Ψ 1 and Ψ 2 in length, respetively. Eah omparison requires O( Σ ) time. In pratie, the time omplexity is O(n Σ ) as eah state needs to e ompared with very few states for the following three reasons. First, the maximum deferment depthψis usually very small. The largest value of Ψ among our 8 primary RE sets in Setion VI is 7. Seond, the length of the deferment hains for most states is muh smaller than Ψ. The largest value of average deferment depth ψ among our 8 RE sets is Finally, many of the state pairs along the deferment hains are not reahale in the merged D 2 FA. Among our 8 RE sets, the largest value of the average numer of omparisons needed is When merging all the D 2 FAs together for an RE set R, the total time required in the worst ase would e O(nΨ 1 Ψ 2 Σ log( R )). The worst ase would happen when the RE set ontains strings and there is no state explosion. In this ase, eah merged D 2 FA would have a numer of states roughly equal to the sum of the sizes of the D 2 FAs eing merged. When there is state explosion, the last D 2 FA merge would e the dominating fator, and the total time would just e O(nΨ 1 Ψ 2 Σ ). When modifying the D2FAMerge algorithm to maintain ak-pointers, the worst ase time would e O(n 2 Σ ) eause we would have to ompare eah state with O(n) other states if none of the andidate pairs are found at a lower level than the state. In pratie, this searh needs to e done for very few states, typially less than 1%.

10 10 The worst ase time omplexity of the final ompression step is the same as that of Kumar et al. s D 2 FA algorithm, whih is O(n 2 (log(n)+ Σ )), sine oth involve omputing a maximum weight spanning tree on the SRG. However, eause we only onsider edges whih improve upon the existing deferment forest, the atual size of the SRG in pratie is typially linear in the numer of nodes. In partiular, for the real-world RE sets that we onsider in the experiments setion, the size of the SRG generated y our final ompression step is on average 100 times smaller than the SRG generated y Kumar et al. s algorithm. As a result the optimization step requires muh less memory and time ompared to the original algorithm. VI. EXPERIMENTAL RESULTS In this setion, we evaluate the effetiveness of our algorithms on real-world and syntheti RE sets. We onsider three variants of our D 2 FA merge algorithm. We denote the main variant as D 2 FAMERGE ; this variant uses our improved D 2 FA onstrution algorithm for one RE. The other two variants are D 2 FAMergeOld, whih uses the D 2 FA onstrution algorithm in [9] to uild D 2 FA for eah RE and was used exlusively in the preliminary version of this paper, and D 2 FAMergeOpt, whih applies our final ompression algorithm after running D 2 FAMERGE. We ompare our algorithms with the original D 2 FA onstrution algorithm proposed in [7] ORIGINAL that optimizes transition ompression and the D 2 FA onstrution algorithm proposed in [15] BACKPTR that enfores the akpointer property desried in Setion V-C. A. Methodology 1) Data Sets: Our main results are ased on eight real RE sets, four proprietary RE sets C7, C8, C10, and C613 from a large networking vendor and four puli RE sets Bro217, Snort 24, Snort31, and Snort 34, that we partition into three groups, STRING, WILDCARD, and SNORT, ased upon their RE omposition. For eah RE set, the numer indiates the numer of REs in the RE set. The STRING RE sets, C613 and Bro217, ontain mostly string mathing REs. The WILDCARD RE sets, C7, C8 and C10, ontain mostly REs with multiple wildard losures.*. The SNORT RE sets, Snort24, Snort31, and Snort34, ontain a more diverse set of REs, roughly 40% of whih have wildard losures. To test salaility, we use Sale, a syntheti RE set onsisting of 26 REs of the form /.* u * l 789!#%&/, where u and l are the 26 upperase and lowerase alphaet letters. Even though all the REs are nearly idential differing only in the harater after the two.* s, we still get the full multipliative effet where the numer of states in the orresponding minimum state DFA roughly doules for every RE added. 2) Metris: We use the following metris to evaluate the algorithms. First, we measure the resulting D 2 FA size (# transitions) to assess transition ompression performane. Our D 2 FAMERGE algorithm typially performs almost as well as the other algorithms even though it uilds up the D 2 FA inrementally rather than ompressing the final minimum state DFA. Seond, we measure the the maximum deferment depth (Ψ) and average deferment depth (ψ) in the D 2 FA to assess how quikly the resulting D 2 FA an e used to perform regular expression mathing. Smaller Ψ and ψ mean that fewer deferment transitions that proess no input haraters need to e traversed when proessing an input string. Our D 2 FAMERGE signifiantly outperforms the other algorithms. Finally, we measure the spae and time required y the algorithm to uild the final automaton. Again, our D 2 FAMERGE signifiantly outperforms the other algorithms. When omparing the performane of D 2 FAMERGE with another algorithm A on a given RE or RE set, we define the following quantities to ompare them: transition inrease is (D 2 FAMERGE D 2 FA size - A D 2 FA size) divided y A D 2 FA size, transition derease is (A D 2 FA size - D 2 FAMERGE D 2 FA size) divided y A D 2 FA size, average (maximum) deferment depth ratio is A average (maximum) deferment depth divided y D 2 FAMERGE average (maximum) deferment depth, spae ratio is A spae divided y D 2 FAMERGE spae, and time ratio is A uild time divided y D 2 FAMERGE uild time. Sine we have a new D 2 FAMERGE algorithm, we needed to rerun our experiments. We ran them on faster proessors than in our onferene version, so all of the algorithms report smaller proessing times than efore. One interesting note is that while the new D 2 FAMERGE performs etter than D 2 FAMergeOld, the running times are essentially the same. 3) Measuring Spae: When measuring the required spae for an algorithm, we measure the maximum amount of memory required at any point in time during the onstrution and then final storage of the automaton. This is a diffiult quantity to measure exatly; we approximate this required spae for eah of the algorithms as follows. For D 2 FAMERGE and D 2 FAMergeOld, the dominant data struture is the D 2 FA. For a D 2 FA, the transitions for eah state an e stored as pairs of input harater and next state id, so the memory required to store a D 2 FA is alulated as = (#transitions) 5 ytes. However, the maximum amount of memory required while running D 2 FAMERGE may e higher than the final D 2 FA size eause of the following two reasons. First, when merging two D 2 FAs, we need to maintain the two input D 2 FAs as well as the output D 2 FA. Seond, we may reate an intermediate output D 2 FA that has more transitions than needed; these extra transitions will e eliminated one all D 2 FA states are added. We keep trak of the worst ase required spae for our algorithm during D 2 FA onstrution. This typially ours when merging the final two intermediate D 2 FA to form the final D 2 FA. For ORIGINAL, we measure the spae required y the minimized DFA and the SRG. For the DFA, the transitions for eah state an e stored as an array of size Σ with eah array entry requiring four ytes to hold the next state id. For the SRG, eah edge requires 17 ytes as oserved in [15]. This leads to a required memory for uilding the D 2 FA of = Q Σ 4+(#edges in SRG) 17 ytes. For D 2 FAMergeOpt, the spae required is the size of the final D 2 FA resulting from the merge step, plus the size of the SRG used y the final ompression algorithm. The sizes are omputed as in the ase of D 2 FAMERGE and ORIGINAL. For BACKPTR, we onsider two variants. The first variant uilds the minimized DFA diretly from the NFA and then sets the deferment for eah state. For this variant, no SRG is

11 11 needed, so the spae required is the spae needed for the minimized DFA whih is Q Σ 4 ytes. The seond variant goes diretly from the NFA to the final D 2 FA; this variant uses less spae ut is muh slower as it stores inomplete transition tales for most states. Thus, when omputing the deferment state for a new state, the algorithm must rereate the omplete transition tales for eah state to determine whih has the most ommon transitions with the new state. For this variant, we assume the only spae required is the spae to store the final D 2 FA whih is = (#transitions) 5 ytes even though more memory is needed at various points during the omputation. We also note that oth implementations must perform the NFA to DFA suset onstrution on a large NFA whih means even the faster variant runs muh more slowly than D 2 FAMERGE. 4) Corretness: We tested orretness of our algorithms y verifying the final D 2 FA is equivalent to the orresponding DFA. Note, we an only do this hek for our RE sets where we were ale to ompute the orresponding DFA. Thus, we only verified orretness of the final D 2 FA for our eight real RE sets and the smaller Sale RE sets. B. D 2 FAMERGE versus ORIGINAL We first ompare D 2 FAMERGE with ORIGINAL that optimizes transition ompression when oth algorithms have unlimited maximum deferment depth. These results are shown in Tale I for our 8 primary RE sets. Tale III summarizes these results y RE group. We make the following oservations. (1) D 2 FAMERGE uses muh less spae than ORIGINAL. On average, D 2 FAMERGE uses 1500 times less memory than ORIGINAL to uild the resulting D 2 FA. This differene is most extreme when the SRG is large, whih is true for the two STRING RE sets and Snort24 and Snort34. For these RE sets, D 2 FAMERGE uses etween 1422 and 4568 times less memory than ORIGINAL. For the RE sets with relatively small SRGs suh as those in the WILDCARD and Snort31, D 2 FAMERGE uses etween 35 and 231 times less spae than ORIGINAL. (2) D 2 FAMERGE is muh faster than ORIGINAL. On average, D 2 FAMERGE uilds the D 2 FA 155 times faster than ORIGINAL. This time differene is maximized when the deferment hains are shortest. For example, D 2 FAMERGE only requires an average of 0.05 mse and 0.09 mse per state for the WILDCARD and SNORT RE sets, respetively, so D 2 FAMERGE is, on average, 247 and 142 times faster than ORIGINAL for these RE sets, respetively. For the STRING RE sets, the deferment hains are longer, so D 2 FAMERGE requires an average of 0.67 mse per state, and is, on average, 35 times faster than ORIGINAL. (3) D 2 FAMERGE produes D 2 FA with muh smaller average and maximum deferment depths than ORIGINAL. On average, D 2 FAMERGE produes D 2 FA that have average deferment depths that are 6.4 times smaller than ORIGINAL and maximum deferment depths that are 4.4 times smaller than ORIGINAL. In partiular, the average deferment depth for D 2 FAMERGE is less than 2 for all ut the two STRING RE sets, where the average deferment depths are 2.15 and Thus, the expeted numer of deferment transitions to e traversed when proessing a length n string is less than n. One reason D 2 FAMERGE works so well is that it eliminates low weight edges from the SRG so that the deferment forest has many shallow deferment trees instead of one deep tree. This is partiularly effetive for the WILDCARD RE sets and, to a lesser extent, the SNORT RE sets. For the STRING RE sets, the SRG is fairly dense, so D 2 FAMERGE has a smaller advantage relative to ORIGINAL. (4) D 2 FAMERGE produes D 2 FA with only slightly more transitions than ORIGINAL, partiularly on the RE sets that need transition ompression the most. On average, D 2 FAMERGE produes D 2 FA with roughly 11% more transitions than ORIGINAL does. D 2 FAMERGE works est when state explosion from wildard losures reates DFA omposed of many similar repeating sustrutures. This is preisely when transition ompression is most needed. For example, for the WILDCARD RE sets that experiene the greatest state explosion, D 2 FAMERGE only has 1% more transitions than ORIGINAL. On the other hand, for the STRING RE sets, D 2 FAMERGE has, on average, 22% more transitions. For this group, ORIGINAL needed to uild a very large SRG and thus used muh more spae and time to ahieve the improved transition ompression. Furthermore, transition ompression is typially not needed for suh RE sets as all string mathing REs an e plaed into a single group and the resulting DFA an e uilt. In summary, D 2 FAMERGE ahieves its est performane relative to ORIGINAL on the WILDCARD RE sets (exept for spae used for onstrution of the D 2 FA) and its worst performane relative to ORIGINAL on the STRING RE sets (exept for spae used to onstrut the D 2 FA). This is desirale as the spae and time effiient D 2 FAMERGE is most needed on RE sets like those in the WILDCARD eause those RE sets experiene the greatest state explosion. (5) Improvement of D 2 FAMERGE over D 2 FAMergeOld. Using our improved algorithm to uild the initial D 2 FAs results in signifiant redution in the final size of the D 2 FA produed y the D 2 FA merge algorithm. On average, D 2 FAMergeOld produes a D 2 FA 8.2% larger than that produed y D 2 FAMERGE. C. Assessment of Final Compression Algorithm We now assess the effetiveness of our final ompression algorithm y omparing D 2 FAMergeOpt to ORIGINAL and D 2 FAMERGE. As expeted D 2 FAMergeOpt produes a D 2 FA that is almost as small as that produed y ORIGINAL; on average, the numer of transitions inreases y only 0.4%. There is a very small inrease for WILDCARD and SNORT eause ORIGINAL also onsiders all edges with weight > 1 in the SRG, whereas D 2 FAMergeOpt does not use edges with weight < 10. There is a signifiant enefit to not using these low weight SRG edges; the deferment depths are muh higher for the D 2 FA produed y ORIGINAL when ompared to the D 2 FA produed y D 2 FAMergeOpt. The final ompression algorithm of D 2 FAMergeOpt does require more resoures than are required y D 2 FAMERGE. In some ases, this may limit the size of the RE set D 2 FAMergeOpt an e used for. However, as explained earlier, D 2 FAMERGE performs est on the WILDCARD (whih has the most state explosion) and performs the worst on the STRING (whih has the no or limited state explosion). So the

12 12 RE set ORIGINAL D 2 FAMERGE # Def. depth RAM Time Def. depth RAM Time States # Trans # Trans Avg. Max. (MB) (s) Avg. Max. (MB) (s) Bro C C C C Snort Snort Snort TABLE I PERFORMANCE DATA OF ORIGINAL AND D 2 FAMERGE RE set D 2 FAMergeOld D 2 FAMergeOpt # Def. depth RAM Time Def. depth RAM Time States # Trans # Trans Avg. Max. (MB) (s) Avg. Max. (MB) (s) Bro C C C C Snort Snort Snort TABLE II PERFORMANCE DATA OF D 2 FAMERGEOLD AND D 2 FAMERGEOPT. D 2 FAMergeOld D 2 FAMERGE D 2 FAMergeOpt RE set group Trans Def. depth ratio Spae Time Trans Def. depth ratio Spae Time Trans Def. depth ratio Spae Time inrease Avg. Max. ratio ratio inrease Avg. Max. ratio ratio inrease Avg. Max. ratio ratio All 20.1% % % STRING 44.0% % % WILDCARD 3.0% % % SNORT 21.3% % % TABLE III COMPARING D 2 FAMERGEOLD, D 2 FAMERGE AND D 2 FAMERGEOPT WITH ORIGINAL. final ompression algorithm is only needed for and is most enefiial for RE sets with limited state explosion. Finally, we oserve that D 2 FAMergeOpt requires on average 113 times less RAM than ORIGINAL, and, on average, runs 9 times faster than ORIGINAL. D. D 2 FAMERGE versus ORIGINAL with Bounded Maximum Deferment Depth We now ompare D 2 FAMERGE and ORIGINAL when they impose a maximum deferment depth ound Ω of 1, 2, and 4. Beause time and spae do not hange signifiantly, we fous only on numer of transitions and average deferment depth. These results are shown in Tale IV. Note that for these data sets, the resulting maximum depth Ψ typially is idential to the maximum depth ound Ω; the only exeption is for D 2 FAMERGE and Ω = 4; thus we omit the maximum deferment depth from Tale IV. Tale V summarizes the results y RE group highlighting how muh etter or worse D 2 FAMERGE does than ORIGINAL on the two metris of numer of transitions and average deferment depth ψ. Overall, D 2 FAMERGE performs very well when presented a ound Ω. In partiular, the average inrease in the numer of transitions for D 2 FAMERGE with Ω equal to 1, 2 and 4, is only 131%, 20% and 1% respetively, ompared to D 2 FAMERGE with unounded maximum deferment depth. Stated another way, when D 2 FAMERGE is required to have a maximum deferment depth of 1, this only results in slightly more than twie the numer of transitions in the resulting D 2 FA. The orresponding values for ORIGINAL are 3121%, 1216% and 197%. These results an e partially explained y examining the average deferment depth data. Unlike in the unounded maximum deferment depth senario, here we see Ω = 1 Ω = 2 Ω = 4 RE set group Trans Avg. def. Trans Avg. def. Trans Avg. dptr der. depth ratio der. depth ratio der. len ratio All 91.3% % % 1.5 STRING 90.0% % % 0.9 WILDCARD 89.3% % % 2.0 SNORT 94.0% % % 1.4 TABLE V COMPARING D 2 FAMERGE WITH ORIGINAL GIVEN MAXIMUM DEFERMENT DEPTH BOUNDS OF 1, 2 AND 4. that D 2 FAMERGE has a larger average deferment depth ψ than ORIGINAL exept for the WILDCARD when Ω is 1 or 2. What this means is that D 2 FAMERGE has more states that defer to at least one other state than ORIGINAL does. This leads to the lower numer of transitions in the final D 2 FA. Overall, for Ω = 1, D 2 FAMERGE produes D 2 FA with roughly 91% fewer transitions than ORIGINAL for all RE set groups. For Ω = 2, D 2 FAMERGE produes D 2 FA with roughly 59% fewer transitions than ORIGINAL for the WILDCARD RE sets and roughly 92% fewer transitions than ORIGINAL for the other RE sets. E. D 2 FAMERGE versus BACKPTR We now ompare D 2 FAMERGE with BACKPTR whih enfores the ak-pointer property desried in Setion V-C. We adapt D 2 FAMERGE to also enfore this ak-pointer property. The results for all our metris are shown in Tale VI for our 8 primary RE sets. We onsider the two variants of BACKPTR desried in Setion VI-A3, one whih onstruts the minimum state DFA orresponding to the given NFA and one whih ypasses the minimum state DFA and goes diretly to the D 2 FA from the given NFA. We note the seond variant appears to use less spae than D 2 FAMERGE. This is partially true sine BACKPTR reates a smaller D 2 FA

13 13 ORIGINAL D 2 FAMERGE RE # Trans Avg. def. depth # Trans Avg. def. depth set Ω = 1 Ω = 2 Ω = 4 Ω=1 Ω=2 Ω=4 Ω = 1 Ω = 2 Ω = 4 Ω=1 Ω=2 Ω=4 Bro C C C C Snort Snort Snort TABLE IV PERFORMANCE DATA FOR ORIGINAL AND D 2 FAMERGE GIVEN MAXIMUM DEFERMENT DEPTH BOUNDS OF 1, 2 AND 4. BACKPTR D 2 FAMERGE with ak-pointer RE Def. depth RAM Time RAM2 Time2 Def. depth RAM Time set # Trans # Trans Avg. Max. (MB) (s) (MB) (s) Avg. Max. (MB) (s) Bro C C C C Snort Snort Snort TABLE VI PERFORMANCE DATA FOR BOTH VARIANTS OF BACKPTR AND D 2 FAMERGE WITH THE BACK-POINTER PROPERTY. than D 2 FAMERGE. However, we underestimate the atual spae used y this BACKPTR variant y simply assuming its required spae is the final D 2 FA size. We ignore, for instane, the spae required to store intermediate omplete tales or to perform the NFA to DFA suset onstrution. Tale VII summarizes these results y RE group displaying ratios for many of our metris that highlight how muh etter or worse D 2 FAMERGE does than BACKPTR. Similar to D 2 FAMERGE versus ORIGINAL, we find that D 2 FAMERGE with the akpointer property performs well when ompared with oth variants of BACKPTR. Speifially, with an average inrease in the numer of transitions of roughly 18%, D 2 FAMERGE runs on average 19 times faster than the fast variant of BACKPTR and 143 times faster than the slow variant of BACKPTR. For spae, D 2 FAMERGE uses on average almost 30 times less spae than the first variant of BACKPTR and on average roughly 42% more spae than the seond variant of BACKPTR. Furthermore, D 2 FAMERGE reates D 2 FA with average deferment depth 2.9 times smaller than BACKPTR and maximum deferment depth 1.9 times smaller than BACKPTR. As was the ase with ORIGINAL, D 2 FAMERGE ahieves its est performane relative to BACKPTR on the WILDCARD RE sets and its worst performane relative to BACKPTR on the STRING RE sets. This is desirale as the spae and time effiient D 2 FAMERGE is most needed on RE sets like those in the WILDCARD eause those RE sets experiene the greatest state explosion. RE set group Trans Def. depth ratio Spae Time Spae2 Time2 inrease Avg. Max. ratio ratio ratio ratio All 17.9% STRING 25.0% WILDCARD 1.3% SNORT 29.7% TABLE VII COMPARING D 2 FAMERGE WITH BOTH VARIANTS OF BACKPTR. F. Salaility results Finally, we assess the improved salaility of D 2 FAMERGE relative to ORIGINAL using the Sale RE set assuming that we have a maximum memory size of 1 GB. For oth ORIGINAL and D 2 FAMERGE, we add one RE at a time from Sale until the spae estimate to uild the D 2 FA goes over the 1GB limit. For ORIGINAL, we are ale only ale to add 12 REs; the final D 2 FA has 397,312 states and requires over 71 hours to ompute. As explained earlier, we inlude the SRG edges in the RAM size estimate. If we exlude the SRG edges and only inlude the DFA size in the RAM size estimate, we would only e ale to add one more RE efore we reah the 1GB limit. For D 2 FAMERGE, we are ale to add 19 REs; the final D 2 FA has 80,216,064 states and requires only 77 minutes to ompute. This data set highlights the quadrati versus linear running time of ORIGINAL and D 2 FAMERGE, respetively. Figure 3 shows how the spae and time requirements grow for ORIGINAL and D 2 FAMERGE as RE s from Sale are added one y one until 19 have een added. RAM (MB) Time (s) Memory required to uild ORIGINAL D 2 FAMERGE #REs 1e Fig Time required to uild ORIGINAL D 2 FAMERGE #REs Memory & time for ORIGINAL s D 2 FA and D 2 FAMERGE s D 2 FA. VII. CONCLUSIONS In this paper, we propose a novel Minimize then Union framework for onstruting D 2 FAs using D 2 FA merging. Our approah requires a fration of memory and time ompared to urrent algorithms. This allows us to uild muh larger D 2 FAs

14 14 than was possile with previous tehniques. Our algorithm naturally supports frequent RE set updates. We onduted experiments on real-world and syntheti RE sets that verify our laims. For example, our algorithm requires an average of 1500 times less memory and 150 times less time than the original D 2 FA onstrution algorithm of Kumar et al.. We also provide an optimization postproessing step that produes D 2 FAs that are essentially as good as those produed y the original D 2 FA onstrution algorithm; the optimization step requires on average 113 times less memory and 9 times less time than the original D 2 FA onstrution algorithm. REFERENCES [1] M. Roesh, Snort: Lightweight intrusion detetion for networks, in Pro. 13th Systems Administration Conferene (LISA), USENIX Assoiation, Novemer 1999, pp [2] Snort, [3] V. Paxson, Bro: a system for deteting network intruders in real-time, Computer Networks, vol. 31, no , pp , [Online]. Availale: iteseer.ist.psu.edu/paxson98ro.html [4] R. Sommer and V. Paxson, Enhaning ytelevel network intrusion detetion signatures with ontext, in Pro. ACM Conf. on Computer and Communiation Seurity, 2003, pp [5] J. E. Hoproft, R. Motwani, and J. D. Ullman, Introdution to Automata Theory, Languages, and Computation. Addison-Wesley, [6] J. E. Hoproft, The Theory of Mahines and Computations. Aademi Press, 1971, h. An nlogn algorithm for minimizing the states in a finite automaton, pp [7] S. Kumar, S. Dharmapurikar, F. Yu, P. Crowley, and J. Turner, Algorithms to aelerate multiple regular expressions mathing for deep paket inspetion, in Pro. SIGCOMM, 2006, pp [8] S. Kumar, J. Turner, and J. Williams, Advaned algorithms for fast and salale deep paket inspetion, in Pro. IEEE/ACM ANCS, 2006, pp [9] C. R. Meiners, J. Patel, E. Norige, E. Torng, and A. X. Liu, Fast regular expression mathing using small tams for network intrusion detetion and prevention systems, in Pro. 19th USENIX Seurity Symposium, Washington, DC, August [10] S. Kumar, B. Chandrasekaran, J. Turner, and G. Varghese, Curing regular expressions mathing algorithms from insomnia, amnesia, and aalulia, in Pro. ACM/IEEE ANCS, 2007, pp [11] R. Smith, C. Estan, and S. Jha, Xfa: Faster signature mathing with extended automata, in Pro. IEEE Symposium on Seurity and Privay, 2008, pp [12] R. Smith, C. Estan, S. Jha, and S. Kong, Deflating the ig ang: fast and salale deep paket inspetion with extended finite automata, in Pro. SIGCOMM, 2008, pp [13] M. Behi and P. Crowley, A hyrid finite automaton for pratial deep paket inspetion, in Pro. CoNext, [14], Extending finite automata to effiiently math perl-ompatile regular expressions, in Pro. ACM CoNEXT, 2008, artile Numer 25. [15], An improved algorithm to aelerate regular expression evaluation, in Pro. ACM/IEEE ANCS, [16] A. V. Aho and M. J. Corasik, Effiient string mathing: an aid to iliographi searh, Communiations of the ACM, vol. 18, no. 6, pp , [17] N. Tuk, T. Sherwood, B. Calder, and G. Varghese, Deterministi memory-effiient string mathing algorithms for intrusion detetion, in Pro. IEEE Infoom, 2004, pp [18] I. Sourdis and D. Pnevmatikatos, Pnevmatikatos: Fast, large-sale string math for a 10gps fpga-ased network intrusion detetion system, in Pro. Int. on Field Programmale Logi and Appliations, 2003, pp [19] L. Tan and T. Sherwood, A high throughput string mathing arhiteture for intrusion detetion and prevention, in Pro. 32nd Annual Int. Symposium on Computer Arhiteture (ISCA), 2005, pp [20] F. Yu, R. H. Katz, and T. V. Lakshman, Gigait rate paket patternmathing using TCAM, in Pro. 12th IEEE Int. Conf. on Network Protools (ICNP), 2004, pp [21] J.-S. Sung, S.-M. Kang, Y. Lee, T.-G. Kwon, and B.-T. Kim, A multigigait rate deep paket inspetion algorithm using tam, in Pro. IEEE GLOBECOM, [22] M. Aliherry, M. Muthuprasanna, and V. Kumar, High speed pattern mathing for network ids/ips, in Pro IEEE International Conferene on Network Protools. Ieee, 2006, pp [23] B. C. Brodie, D. E. Taylor, and R. K. Cytron, A salale arhiteture for high-throughput regular-expression pattern mathing, SIGARCH Computer Arhiteture News, [24] M. Behi and P. Crowley, Effiient regular expression evaluation: Theory to pratie, in Pro. ACM/IEEE ANCS, [25] S. Kong, R. Smith, and C. Estan, Effiient signature mathing with multiple alphaet ompression tales, in Pro. 4th Int. Conf. on Seurity and privay in ommuniation netowrks (SeureComm). ACM Press, 2008, p. 1. [26] M. Behi and S. Cadami, Memory-effiient regular expression searh using state merging, in Pro. INFOCOM. IEEE, [27] F. Yu, Z. Chen, Y. Diao, T. V. Lakshman, and R. H. Katz, Fast and memory-effiient regular expression mathing for deep paket inspetion, in Pro. ACM/IEEE Symposium on Arhiteture for Networking and Communiations Systems (ANCS), 2006, pp [28] L. Yang, R. Karim, V. Ganapathy, and R. Smith, Fast, memory-effiient regular expression mathing with NFA-OBDDs, Computer Networks, vol. 55, no. 55, pp , [29] R. Sidhu and V. K. Prasanna, Fast regular expression mathing using fpgas, in Pro. IEEE Symposium on Field-Programmale Custom Computing Mahines FCCM, 2001, pp [30] C. R. Clark and D. E. Shimmel, Effiient reonfigurale logi iruits for mathing omplex network intrusion detetion patterns, in Pro. Field-Programmale Logi and Appliations, 2003, pp [31], Salale pattern mathing for high speed networks, in Pro. 12th Annual IEEE Symposium on Field-Programmale Custom Computing Mahines (FCCM), Washington, DC, [32] I. Sourdis and D. Pnevmatikatos, Pre-deoded ams for effiient and high-speed nids pattern mathing, in Pro. Field-Programmale Custom Computing Mahines, [33] A. Mitra, W. Najjar, and L. Bhuyan, Compiling PCRE to FPGA for aelerating SNORT IDS, in Pro. ACM/IEEE ANCS, [34] J. Mosola, J. Lokwood, R. P. Loui, and M. Pahos, Implementation of a ontent-sanning module for an internet firewall, in Pro. IEEE Field Programmale Custom Computing Mahines, [35] A. Bremler-Barr, D. Hay, and Y. Koral, Compatdfa: Generi state mahine ompression for salale pattern mathing, in Pro. IEEE INFOCOM. Ieee, 2010, pp [36] Y. Liu, L. Guo, M. Guo, and P. Liu., Aelerating DFA onstrution y hierarhial merging, in Pro. IEEE 9th Int. Symposium on Parallel and Distriuted Proessing with Appliations, Jignesh Patel Jignesh Patel is urrently a Ph.D. student in the Department of Computer Siene and Engineering at Mihigan State University. His researh interests inlude algorithms, networking, and seurity. Alex X. Liu reeived his Ph.D. degree in omputer siene from the University of Texas at Austin in He reeived the IEEE & IFIP William C. Carter Award in 2004 and an NSF CAREER award in He reeived the Withrow Distinguished Sholar Award in 2011 at Mihigan State University. His researh interests fous on networking, seurity, and dependale systems. Eri Torng Eri Torng reeived his Ph.D. degree in omputer siene from Stanford University in He is urrently an assoiate professor and graduate diretor in the Department of Computer Siene and Engineering at Mihigan State University. He reeived an NSF CAREER award in His researh interests inlude algorithms, sheduling, and networking.