Collective Biobjective Optimization Algorithm for Parallel Test Paper Generation

Transcription

1 Proceeings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 1) Collective Biobjective Optimization Algorithm for Parallel Test Paper Generation Minh Luan Nguyen Data Analytics Department Institute for Infocomm Research Singapore Siu Cheung Hui School of Computer Engineering Nanyang Technological University Singapore Alvis C. M. Fong School of Computing Science University of Glasgow Scotlan, UK Abstract Parallel Test Paper Generation (k-tpg) is a biobjective istribute resource allocation problem, which aims to generate multiple similarly optimal test papers automatically accoring to multiple userspecifie criteria. Generating high-quality parallel test papers is challenging ue to its NP-harness in maximizing the collective objective functions. In this paper, we propose a Collective Biobjective Optimization () algorithm for solving k-tpg. is a multi-step greey-base approximation algorithm, which exploits the submoular property for biobjective optimization of k-tpg. Experiment results have shown that has rastically outperforme the current techniques in terms of paper quality an runtime efficiency. 1 Introuction With the rapi growth of the Internet an mobile evices, Webbase eucation has become a ubiquitous learning platform in many institutions to provie stuents with online learning courses an materials through freely accessible eucational websites such as Khan Acaemy, or online classes such as Coursera an Uacity. To make learning effective, it is important to assess the proficiency of the stuents while they are learning the concepts. As there may have many stuents in an online class [Martin, 1], to ensure the assessment reliability of large-scale Web-base testing, peagogical practitioners have suggeste that it is necessary to compose multiple tests from a large question pool with equivalent properties. One promising approach to support large-scale Web-base testing is parallel test paper generation (k-tpg), which generates k similarly optimal test papers automatically accoring to a user specification base on multiple criteria. Specifically, it aims to fin k isjoint subsets of questions from a question atabase to form multiple test papers accoring to a user specification base on total time, topic istribution, ifficulty egree, iscrimination egree, an so on. k-tpg is a challenging problem especially with large number of questions an large number of generate test papers (i.e. large k) ue to its NP-harness. Different from other multiobjective optimization problems, k-tpg has collective objective functions to be efine base on the evaluation objectives of k generate test papers instea of a single test paper. In fact, k-tpg is close to the spirit of optimal istribute allocation problems [Feige an Vonrak, ] with collective objective functions. Formally, k-tpg is a biobjective optimal istribute resource allocation problem, which aims to simultaneously maximize two objective functions uner a multiimensional Knapsack constraint. The first objective can be formulate as a Welfare Allocation problem [Vonrak, ], which aims to maximize the total quality of the generate test papers. The secon objective can be formulate as a Fairness Allocation problem [Asapour an Saberi, 1; Bertsimas et al., 11], which aims to maximize the fairness quality of the generate test papers. Traitionally, optimizing these two objectives is NP-har an consiere separately. To the best of our knowlege, there has been no attempt to solve both of the objectives simultaneously. Submoular-base approximation algorithms, which are polynomial time algorithms that have performance guarantees, have been stuie for istribute optimization problems [Neic et al., 1]. Inspire by this novel iea, we propose a greey-base approximation algorithm, calle Collective Biobjective Optimization (), for k-tpg. The key iea of is twofol. Firstly, it analyzes the properties of the two collective objectives an reformulates the k-tpg problem such that it can be solve effectively. Seconly, we propose an effective multi-step framework for the reformulate optimization problem. More importantly, we exploit submoular optimization techniques to esign greey-base approximation algorithms for enhancing each step in the propose framework. In this paper, we will iscuss the propose algorithm for k-tpg an its performance evaluation. Relate Work For the past few years, heuristic-base intelligent techniques such as Tabu Search [Hwang et al., ], Particle Swarm Optimization [Ho et al., ] an Ant Colony Optimization [Hu et al., 9] have been propose for k-tpg. However, the quality of the generate parallel test papers is often unsatisfactory [Hwang et al., ; Ho et al., ; Hu et al., 9] accoring to users test paper specifications. The main issue of the current techniques is the exhaustive search to optimize the biobjective functions simultaneously in the very large space of possible caniates with multicriteria 1

2 constraints. Although these heuristic-base techniques are straightforwar to implement, they suffer from some rawbacks. They ten to get stuck in a local optimum as they are mainly base on traitional weighting parameters for multiobjective optimization [Ishibuchi et al., ]. 3 Parallel Test Paper Generation Let Q = {q 1, q,.., q n } be a ataset consisting of n questions, C = {c 1, c,.., c m } be a set of m ifferent topics, an Y = {y 1, y,.., y k } be a set of k ifferent question types. Each question q i Q, where i {1,,.., n}, has attributes A = {q, e, t,, c, y} efine as follows: Question q: It is use to store the question ientity. Discrimination egree e: It is a positive integer value to istinguish user proficiency. Question time t: It is a positive integer value to inicate the average time in minutes neee to answer. Difficulty egree : It is a positive integer value to inicate the question ifficulty. Relate topic c: It is a set of relate topics of a question. Question type y: It is the type of a question. A test paper specification S = N, T, D, C, Y consists of attributes incluing the number of questions N, total time T, average ifficulty egree D, topic istribution C = {(c 1, pc 1 ), (c, pc ),.., (c M, pc M )} an question type istribution Y = {(y 1, py 1 ), (y, py ),.., (y K, py K )}, which are efine base on the attributes of the selecte questions. In 1-TPG, it selects an optimal subset of questions to form a test paper P, which maximizes the average iscrimination egree f(p ) = 1 N N e i, q i P while satisfying multiple user-specifie criteria in a test paper specification S P = S. In k-tpg, user specification is given as a pair of k, S, where k is an integer parameter in aition to the test paper specification S. The k-tpg aims to select k isjoint subsets of questions P 1, P,.., P k from the question ataset Q to form k test papers with specification S Pi, which satisfies the test paper specification such that S Pi = S, i [1, k]. Apart from maximizing the objective function as in single test paper generation, k-tpg aims to generate the k test papers with similar optimal quality accoring to the specification. It aims to provie similar quality of test papers for ifferent users. As such, there are two important collective objective functions that nee to be maximize in k-tpg: total quality maximization an fairness quality maximization. Total Quality Maximization. It aims to maximize the sum of iscrimination egrees of k test papers k f(p i), where f(p i ) is the iscrimination egree of test paper P i, i [1, k]. Formally, the k-tpg problem is state as follows: maximize P 1,..,P k f(p i ) (1) Fairness Quality Maximization. It aims to maintain the fairness among the generate test papers with equivalent iscrimination egrees. However, solely maximizing the total quality objective oes not guarantee the fairness quality. Accoring to [Asapour et al., ; Bertsimas et al., 11], it is more effective to optimize a fair allocation of the iscrimination egrees of k generate test papers as: maximize P 1,..,P k min f(p i) () 1 i k subject to the local multicriteria constraints: P i P j =, i j (3) S Pi = S () Propose Approach To simultaneously optimize the total quality an fairness quality objective functions is a challenging problem. However, by exploiting the relationship between the two objective functions base on the submoular property [Lovasz, 193], we can evise an effective an efficient approach..1 Problem Reformulation Before reformulating the k-tpg problem, we review the funamental concept of submoular function in optimization. Definition 1 (Submoular Function Optimization). Given a set X, a non-negative function f : X R +, T X, an x X. Let f(s) x = f (x S) = f(s {x}) f(s) be the ecreasing marginal value of f at S with respect to x. A non-negative function f : X R + is submoular if for every set S, T X, we have: f(s T ) + f(s T ) f(s) + f(t ) Although NP-har, the submoular property has le to the existence of a Greey-base Approximation Algorithm [Fujishige, ] for the maximization. It is wiely known that a linear function as well as the combination of linear functions are aslo submoular [Fujishige, ]. [Nguyen et al., 13] shows that 1-TPG can be reformulate as an integer linear program which aims to maximize a linear objective function f(p ) uner multiple knapsack constraints. Hence, the quality function f(p ) is submoular ue to the linearity property. Lemma 1. Given a test paper P generate from a question set Q (P Q), the quality evaluation function of iscrimination egree f(p ) : Q R + is submoular an monotone. Hence, the total quality objective function is also submoular ue to a linear combination of k submoular functions. Corollary 1. The total quality objective function of k test papers k f(p i), P i Q is submoular an monotone. To jointly optimize the biobjective functions, we take the avantage of submoularity to reformulate the total quality objective such that it also integrates the fairness objective. Let f φ (P ) = min{f(p ), φ} be a truncate function, where φ is a non-negative constant. Note that f φ (P ) is also submoular an monotone ue to [Fujishige, ]. For any constant φ, we have the following important observation: min f(p l) φ l {1,..,k} f φ (P l ) = kφ () As k is a constant, this means that the fairness quality objective value of a paper is larger than or equal to φ if the total quality objective value is kφ an vice versa. Therefore, we have the following result in Theorem 1: 19

3 Theorem 1. Given the optimal fairness value φ, i.e., max P1,..,P k min l f(p l ) = φ, then it is sufficient to jointly optimize the two objective functions of the original k-tpg problem by solving the following equivalent problem: g-17pt maximize P 1,..,P k f φ (P l ) s.t. P i P j =, S Pi = S () i j Proof Sketch: The original k-tpg problem is equivalent to the problem given in () since, by (), there exists k test papers P 1,.., P k such that k f φ (P l) = kφ. In aition, because of the property of the truncate function f φ (P l ), we have max k f φ (P l) kφ. Thus, for any optimal generate papers P 1,.., P k of the problem in (), it must satisfy that k f φ (P l) = kφ. Hence, we have min l f(pl ) = φ ue to ().. Collective Biobjective Optimization In this paper, we propose a novel Collective Biobjective Optimization () approach for k-tpg. We observe that the reformulate problem () can be solve effectively by an iterative multi-step framework. More importantly, we exploit the submoular optimization technique to esign greey-base approximation algorithm for enhancing the steps in the propose approach. Figure 1 shows the main steps of the approach: Optimal Fairness Value Estimation, Infeasible Allocation Detection, Total Quality Maximization, Fairness Quality Balancing an Local Constraint Improvement. In the first step, after each feeback loop, either: (1) reuces half of the search space on φ if Infeasible Allocation Detection fails or () fins a better solution in terms of both of the objectives. In the remaining steps, will generate k test papers progressively by using the greey-base algorithm for 1-TPG an ajusting the fairness value φ. Due to NPharness of 1-TPG, when φ approaches its optimal value, we can only achieve a fraction β 1 of the optimal objective value of φ, where β is a constant that will be etermine in Section.. As such, our goal is to allocate questions into k test papers such that for all k papers, we have f φ (P l ) βφ, l = 1..k..3 Optimal Fairness Value Estimation In each iteration, a new set of k papers will be generate accoring to a new fairness value φ. This step aims to fin a new optimal value of φ to continue the co-optimization process. Generally we o not know the optimal value φ. Therefore, we nee to search for it using the binary search strategy, starting with the interval [φ min, φ max ] = [, N max q Q f(q)]. In each step, we test the center φ = (φ min + φ max )/ of the current interval [φ min, φ max ] of possible fairness value.. Infeasible Allocation Detection This step aims to etect whether it is possible to generate k test papers such that f φ (P l ) βφ, l = 1..k. Specifically, it checks whether the estimate value of φ is appropriate or not. To overcome this, we propose a greey-base approximation Figure 1: Propose Approach Algorithm 1: Infeasible Allocation Detection Process: 1 P l for all l = 1..k; for to k*n o foreach question q Q \ (P 1... Pk ) an l {1..k} o 3 if q satisfies the carinality constraint an local constraints P l then Compute the improvement: Ψ l,q w(p l + {q}) w(p l ) ; (l, q ) argmax (l,q) Ψ l,q ; P l P t {q }; 7 ξ = k f φ(p l ); if ξ < 1 kφ then return Yes else return No algorithm that generalizes the basic greey-base approximation algorithm for submoular function. For the sake of simplicity, we consier only the topic an question type constraints. As such, the result obtaine is the upper boun of the actual approximate solution. However, it is sufficient for the early infeasible allocation etection purpose. The algorithm is outline in Algorithm 1. In each iteration, we first check that whether the question q can be possibly allocate to a paper P t. To verify whether q can be possibly allocate to P t, we first check whether P t is full, i.e., P t = N. Then, we check whether allocating q to P t will violate the two constraints. If yes, we then compute the marginal improvement value Ψ t,q of allocating question q into P t. Next, we greeily choose the best allocation of test paper an question (P t, q ) with the maximum Ψ t,q value. We provie a theoretical result of this step. For general cases, i.e., k, we can prove that it achieves a weaker approximation ratio of 1. Theorem. Consier only the content constraints with k, Algorithm 1 can obtain a set of generate test papers P 1,.., P k with the total quality value such that: f φ (P i ) 1 max P 1,..,P k f φ (P i ) = 1 OP T

4 where OPT refers to the optimal value. Proof Sketch: Let H be the original problem of allocating k N out of n questions to k papers P 1,.., P k such that it maximizes the total quality objective k f φ(p i ). Let H be the problem on the n 1 remaining questions after the first question q t is selecte for paper P j. On the problem H, the quality evaluation function f j (P j ) = f φ (P j ) is replace by f j (P j) = f j (P j {q t }) f j ({q t }). All other quality evaluation function f i (P i ), i j, are unchange. Let V AL(H) be the value of the allocation prouce by Algorithm 1 an OP T (H) be the value of the optimal allocation. Let p = f j ({q t }). By efinition of H, it is clear that V AL(H) = V AL(H ) + p. By submoularity, we can show that OP T (H) OP T (H ) + p. This proof is complete by inuction on H since H is also a submoular function: OP T (H) OP T (H )+p V AL(H )+p = V AL(H) Theorem means that Algorithm 1 can achieve 1/- approximation ratio for the total quality maximal objective base on a given fairness value φ. Base on this, we can ecie whether it is feasible to allocate k papers corresponing to the current fairness φ.. Total Quality Maximization It progressively solves k problem instances of 1-TPG to generate k papers P 1, P,.., P k. Base on the submoular property of f φ (P ), we greeily select questions that maximize the objective function while paying attention to satisfying the multiple knapsack constraints. This is motivate by an algorithm that maximizes a submoular function uner a knapsack constraint: max f(s) s.t. c(x) b S x S where c(x) is a cost function an b R is a buget constraint. Svirienko [Svirienko, ] propose the greeybase algorithm for solving this problem with an approximation ratio of 1 1 e. It efines the marginal gain f (x S) c(x) when aing an item x into the current solution S as a ratio between the iscrete erivative f (x S) an the cost c(x). This algorithm starts with S =, an then iteratively as the element x that maximizes the marginal gain ratio among all elements that satisfy the remaining buget constraint: (x S i ) S i+1 = S i { arg max } x V \S i:c(x) b c(s i) c(x) We exten the above algorithm of submoular function maximization uner a knapsack constraint for solving the case of multiple knapsack constraints. Algorithm gives the greeybase algorithm for this step. This algorithm maintains a set of weights that are incrementally ajuste in a multiplicative manner. These weights capture the possibility that each constraint is close to be violate when maximizing the objective function of generating a paper. The algorithm is base on a main loop in Line. In each iteration, the algorithm selects an available question that maximizes the sum of marginal gain ratio normalize by the weights as follows: P i+1 l = P i l {arg max q j Q m fφ (q j P i l ) A ij h i } Algorithm : Total Quality Maximization Process: 1 for to k o P l ; 3 for to m o h i = 1/b i ; while m q j arg max q j Q P l P l {q j }; b ih i µ an P l N o m fφ (q j P l ) A ijh i ; 7 for to m o h i h i µ A ij b i ; Q Q \ {q j } 9 return P 1, P,.., P k where A is a matrix an b is the vector of multiple knapsack constraints. Here, fφ (q j P l ) is the submoular incremental marginal value of question q j to the paper P l. The parameter µ in Algorithm is important as it ensures that multiple knapsack constraints will not be violate while maximizing the submoular objective function. Let H = min{b i /A ij : A ij > } be the with of the knapsack constraints. By setting the parameter µ = e H m, Algorithm can achieve near-optimal results. [Azar an Gamzu, 1] has prove the following theoretical result: Lemma. Algorithm (Line -) attains an approximation guarantee such that f φ (P i ) (1 1 e )f φ(p i ) f φ (P i ) > 1 f φ(p i ) where Pi refers to the optimal solution of each main loop. Corollary. The total quality objective function of k test papers P 1,.., P k attaine by Algorithm satisfies: f φ (P i ) 1 max f φ (Pi ) = 1 OP T P 1,..,P k Proof Sketch: Let Pi be the optimal solution corresponing to each main loop of Algorithm. By using Lemma, we can irectly erive the result.. Fairness Quality Balancing This step aims to employ a novel fairness balancing strategy to improve the fairness quality objective. Recall that our goal is to allocate questions into k test papers such that for all papers, we have f φ (P l ) βφ, l = 1..k. However, this is not easy to achieve especially when φ approaches the optimal value. This is because in the small an less abunant pool of questions, Algorithm greeily selects all questions with high value f φ (q) for some of the generate papers P 1, P,.., P s to satisfy f φ (P l ) βφ, l = 1..s. As a result, the subsequent papers P s+1,.., P k might not have enough goo questions to satisfy this requirement. We nee to swap some of the questions from the satisfie papers with questions from unsatisfie papers to achieve a fair allocation. We will move questions from satisfie papers to unsatisfie papers an vice versa, until all papers satisfy the requirement f φ (P l ) βφ, l = 1..k. Let s efine the swapping operation 1

5 Algorithm 3: Fairness Quality Balancing Process: while i, j : f φ (P i ) 3βφ; f φ (P j ) < βφ o 1 foreach q i P i o P j {P j \ {q j }} {q i }; 3 P i {P i \ {q i }} {q j }; if f φ (P j ) βφ then break; return P 1, P,.., P k as follows. Select a satisfie paper P i = {q1, i q, i...qn i } for which f φ (P i ) 3βφ. Such a paper is always ensure through appropriate choice of α an β as shown in Lemma 3. Then, we select an unsatisfie paper P j, i.e., f φ (P j ) βφ. In the paper P i, choose t < N such that f φ ({q1, i q, i...qt 1}) i < βφ, f φ ({q1, i q, i...qt}) i βφ, an f φ ({qt}) i < βφ. Let Λ i = {q1, i q, i...qt}. i As f φ ( ) =, the set Λ i is not empty. In the reverse irection, let Λ j = {q j 1, qj,...qj t } be a set of questions in P j such that each pair of questions ql i, l = 1..t, an q j l, l = 1..t, has the same topic an question type. We reallocate the questions of papers P i an P j by swapping questions of the two sets Λ i an Λ j. We note that the swapping operation oes not violate the multiple knapsack constraints. More importantly, it improves the fairness among all papers. The swapping operation is given in Algorithm 3. By some analysis, we can prove that swapping questions oes not ecrease the value of f φ (P i ) by more than βφ. Thus, P i is still satisfie. Moreover, the previously unsatisfie test paper becomes satisfie by this operation. We can also make sure that we can always perform this operation until all test papers are satisfie by choosing β = α/3, where α = 1/ is the approximation ratio of Algorithm 1. Lemma 3. If we choose β = α/3 = 1/, it is guarantee that after at most t swapping operations, all test papers will be satisfie, i.e., f φ (P l ) βφ, l = 1..k. Lemma 3 ensures that there always exists a test paper such that f φ (P l ) 3βφ. It also ensures that all test papers will be satisfie after at most t swaps..7 Local Constraint Improvement So far, we have achieve a set of k test papers, having guarantee on the biobjective values an satisfying the multiple knapsack constraints. However, these test papers have not yet satisfie the total time an ifficulty egree constraints. Here, we propose an effective metho for satisfying these constraints. Given a test paper P = P l = {q 1, q,.., q N }, l = 1..k, let q = {, 1} n, q = N be the binary representation of the initial solution P. From the previous step, we have Aq b, which is the -1 ILP formulation of the corresponing 1-TPG problem. This step aims to turn the existing solution q to q 1 such that Aq 1 = b. This is the Subset Vector Sum problem, which is NP-har. Here, we only nee to optimize the total time an ifficulty egree constraints. Let n a ilq b i be the total time constraint an n a jlq b j be the ifficulty egree constraint. To optimize constraint satisfaction while ensuring the objective value f(p l ) will not ecrease, we reformulate the 1-TPG problem #Questions #Topics #Question Types Distribution D 1 3 uniform D 3 3 normal D 3 3 uniform D 3 normal Table 1: Test Datasets to unconstraine submoular optimization by introucing two weighting parameters λ 1 an λ : n n f ξ (P ) = f(p ) λ 1 a il q b i λ a jl q b j It is not ifficult to show that the problem max f ξ (P l ) is an unconstraine non-monotone submoular maximization, which can be solve by an effective eterministic approximation local search propose by Feige et al. [Feige et al., 7].. Termination The process is repeate until the termination conition is reache: φ max φ min δ, where δ is a preefine threshol. An optimal value of δ =. was foun experimentally..9 Theoretical Analysis We summarize the approximation results of the propose approach for biobjective optimization of k-tpg. Theorem 3. The propose approach has achieve the following theoretical biobjective approximation results of the total quality maximization an fairness quality maximization: f(p i ) 1 max P 1,..,P k min..k f(p i ) = 1 OP T f(p l) 1 max min f(p P 1,..,P k l )..k Performance Evaluation The performance of is compare with other techniques incluing the following 3 re-implemente algorithms for k- TPG: k-ts [Hwang et al., ], k-pso [Ho et al., ] an k-aco [Hu et al., 9] base on the publishe articles. As there is no benchmark ata available, we generate large-size synthetic atasets, namely D 1, D, D 3 an D, for performance evaluation. In atasets D 1 an D 3, the value of each attribute is generate accoring to a uniform istribution. On the other han, in atasets D an D, the value of each attribute is generate accoring to a normal istribution. Table 1 summarizes the atasets. We analyze the quality an runtime efficiency of the approach for k-tpg base on the large-scale atasets by using ifferent specifications. Here, we have esigne 1 test specifications in the experiments. We vary the parameters in orer to have ifferent test criteria in the test specifications. The number of topics is specifie between an. The total time is set between an minutes, an it is also set proportional to the number of selecte topics for each specification. The average ifficulty egree is specifie ranomly between 3 an 9. We have conucte the experiments accoring to ifferent values of k, i.e., k = 1,, 1, 1,.

6 Average Time (in m) Average Time (in m) Average Time (in m) Average Time (in m) k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers (a) Dataset D 1 (b) Dataset D (c) Dataset D 3 Figure : Performance Results Base on Average Runtime To evaluate the quality for k-tpg, we use the common metrics incluing the Average Discrimination Degree an Average Constraint Violation for 1-TPG [Nguyen et al., 13] base on the same user specification S. From the common metrics, for a given ataset D an k, we efine the following metrics for k-tpg base on the average an eviation of the quality of generate test papers: the Average Discrimination Degree M k,d, the Deviate Discrimination Degree V k,d, the Average Constraint Violation M k,d c an the Deviate Constraint Violation Vc k,d. Specifically, given a ataset D an for each k, we generate 1 test paper sets P 1, P,..., P k from the 1 specifications. For each of the 1 test paper sets, we measure the mean an eviation of quality of k test papers accoring to the iscrimination egree an constraint violation. Finally, we obtain the performance results base on the measures by averaging the means an eviations over the 1 test paper sets. For high quality parallel test papers, the Average Discrimination Degree shoul be high an the Average Constraint Violation shoul be small. Similarly, the Deviate Discrimination Degree an the Deviate Constraint Violation shoul be small..1 Performance Results base on Runtime Figure compares the runtime performance of the techniques base on the atasets. The results have clearly shown that the propose approach significantly outperforms the other heuristic techniques in runtime for the ifferent atasets. generally requires less than minutes to complete the parallel paper generation process. Moreover, the propose approach is quite scalable in runtime on ifferent ataset sizes an istributions. In contrast, the other techniques are not efficient to generate high quality parallel test papers. Particularly, the runtime performance of these techniques egraes quite baly as the ataset size or the number of generate parallel test papers gets larger, especially for imbalance atasets D an D.. Performance Results base on Quality Figure 3 shows the quality performance results of the techniques base on the Mean Discrimination Degree M k,d an Deviate Discrimination Degree V k,d. As can be seen from Figure 3, has consistently achieve higher Mean Discrimination Degree M k,d an lower Deviate Discrimination Degree V k,d than the other heuristic k-tpg techniques for the generate parallel test papers. Particularly, can generate high quality test papers with M k,d 7. Note that the lower () Dataset D Deviate Discrimination Degree V k,d value inicates that the generate parallel test papers have similar quality in terms of iscrimination egree. Generally, for a specific value of k, we observe that the quality of the generate parallel test papers of all techniques base on the Mean Discrimination Degree M k,d an the Deviate Discrimination Degree V k,d ten to be improve when the ataset size gets larger. Figure gives the quality performance results of the techniques base on the Mean Constraint Violation M D,k c an Deviate Constraint Violation Vc D,k. We also observe that has consistently outperforme the other techniques on Mean Constraint Violation M D c an Deviate Constraint Violation Vc D,k base on the atasets. The Mean Constraint Violation of tens to ecrease whereas the Mean Constraint Violations of the other 3 techniques increase quite fast when the ataset size or the number of specifie constraints gets larger. In particular, can generate high quality parallel test papers with M D,k c 1 for all atasets. Also, is able to generate higher quality parallel test papers on larger atasets while the other techniques generally egrae on the quality of the generate test papers when the ataset size gets larger. In aition, we fin that when k gets larger, the Mean Constraint Violation M k,d c on a specific ataset D tens to ecrease. Similarly, the Deviate Constraint Violation Vc k,d quality tens to increase when k gets larger. These results have shown that the quality base on constraint violation on a specific ataset D tens to egrae when k gets larger. The goo performance of is ue to main reasons. Firstly, as is an approximation algorithm with constant performance guarantee, it can fin the near-optimal solution for objective functions of k-tpg effectively an efficiently while satisfying the multiple constraints without using weighting parameters. As such, can achieve better paper quality an runtime efficiency as compare with other heuristic-base k-tpg techniques. Seconly, is a submoular greeybase algorithm, which is able to prouce goo solution in efficient polynomial runtime. Thus, can also improve its computational efficiency on large-scale atasets as compare with the other k-tpg techniques. Conclusion In this paper, we have propose an effective an efficient Collective Biobjective Optimization algorithm for solving k-tpg. The key success of lies in the synergy of the effective problem reformulation with the exploitation of submoular property for collective biobjective optimization. incorporates submoular optimization mechanism an submoular 3

7 Mean & Deviate Discrimination Degree 1 k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers Mean & Deviate Discrimination Degree 1 k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers Mean & Deviate Discrimination Degree 1 k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers Mean & Deviate Discrimination Degree 1 k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers (a) Dataset D 1 (b) Dataset D (c) Dataset D 3 () Dataset D Figure 3: Results on Quality base on Mean Discrimination Degree M k,d an Deviate Discrimination Degree V k,d k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers k=1 k= k=1 k=1 k= Number of Parallel Generate Test Papers (a) Dataset D 1 (b) Dataset D (c) Dataset D 3 () Dataset D Figure : Results on Quality base on Mean Constraint Violation M k,d c an Deviate Constraint Violation Vc k,d fairness balancing to jointly optimize the total quality maximization an the fairness quality maximization objectives. To the best of our knowlege, is a pioneering greeybase approximation algorithm, which can achieve provably near-optimal solutions in polynomial runtime for k-tpg. The performance results on various atasets have shown that the approach has achieve generate parallel test papers with not only high quality, but also runtime efficiency when compare with other k-tpg techniques. References [Asapour an Saberi, 1] A. Asapour an A. Saberi. An approximation algorithm for max-min fair allocation of inivisible goos. SIAM Journal on Computing, 39(7):97 99, 1. [Asapour et al., ] A. Asapour, U. Feige, an A. Saberi. Santa claus meets hypergraph matchings. Approximation, Ranomization an Combinatorial Optimization. Algorithms an Techniques, pages 1,. [Azar an Gamzu, 1] Yossi Azar an Iftah Gamzu. Efficient submoular function maximization uner linear packing constraints. In Automata, Languages, an Programming, pages 3. Springer, 1. [Bertsimas et al., 11] D. Bertsimas, V. F. Farias, an N. Trichakis. The price of fairness. Operations research, 9(1):17 31, 11. [Feige an Vonrak, ] U. Feige an J. Vonrak. Approximation algorithms for allocation problems: Improving the factor of 1-1/e. In 7th Annual IEEE Symposium on Founations of Computer Science, pages 7 7,. [Feige et al., 7] U. Feige, V.S. Mirrokni, an J. Vonrak. Maximizing non-monotone submoular functions. SIAM Journal on Computing, (): , 7. [Fujishige, ] S. Fujishige. Submoular functions an optimization. Elsevier Science,. [Ho et al., ] T. F. Ho, P. Y. Yin, G. J. Hwang, S. J. Shyu, an Y. N. Yean. Multi-objective parallel test-sheet composition using enhance particle swarm optimization. Journal of Eucational Technology an Society, 1():193,. [Hu et al., 9] X. M. Hu, J. Zhang, H. S. H. Chung, O. Liu, an J. Xiao. An intelligent testing system embee with an ant-colony-optimization-base test composition metho. IEEE Transactions on Systems, Man, an Cybernetics, 39():9 9, 9. [Hwang et al., ] G. J. Hwang, H. C. Chu, P. Y. Yin, an J. Y. Lin. An innovative parallel test sheet composition approach to meet multiple assessment criteria for national tests. Computers an Eucation, 1(3):1 17,. [Ishibuchi et al., ] H. Ishibuchi, N. Tsukamoto, an Y. Nojima. Evolutionary many-objective optimization: A short review. In IEEE Worl Congress on Evolutionary Computation, pages 19,. [Lovasz, 193] L. Lovasz. Submoular functions an convexity. Mathematical programming: the state of the art, pages 7, 193. [Martin, 1] F. G. Martin. Will massive open online courses change how we teach? Communication of the ACM, ():, August 1. [Neic et al., 1] A. Neic, A. Ozaglar, an P. A. Parrilo. Constraine consensus an optimization in multi-agent networks. IEEE Transactions on Automatic Control, ():9 93, 1. [Nguyen et al., 13] M. L. Nguyen, S. C. Hui, an A. C. M. Fong. Large-scale multiobjective static test generation for web-base testing with integer programming. IEEE Transactions on Learning Technologies, (1): 9, 13. [Svirienko, ] M. Svirienko. A note on maximizing a submoular set function subject to a knapsack constraint. Operations Research Letters, 3(1):1 3,. [Vonrak, ] J. Vonrak. Optimal approximation for the submoular welfare problem in the value oracle moel. In Proceeings of the th annual ACM symposium on Theory of computing, pages 7 7,.