Knowledge Representation N = f0; 1; 2; 3;:::g. A computer program p for a function f : N! N represents knowledge of how to compute f. p may represent

Size: px

Start display at page:

Download "Knowledge Representation N = f0; 1; 2; 3;:::g. A computer program p for a function f : N! N represents knowledge of how to compute f. p may represent"

Howard Cameron
5 years ago
Views:

1 Complexity and Information Deciencies of Learned Knowledge Representations (Preliminary Results) John Case University of Delaware Keh-Jian Chen Academia Sinica Sanjay Jain National U. of Singapore Jim Royer Syracuse University 1

2 Knowledge Representation N = f0; 1; 2; 3;:::g. A computer program p for a function f : N! N represents knowledge of how to compute f. p may represent additional knowledge, perhaps implicit and needing to be safely extracted. 2

3 Gold-Style Learning Theory f(0); f(1); : : : In,! M,! Out p0; p1; : : : ; j p t ; : : : Ref: S. Jain, D. Osherson, J. Royer, and A. Sharma, Systems That Learn, 2nd edition, MIT Press, Think of p 0 ;p 1 ;::: as M's succession of not-necessarily-rational beliefs about how to compute f based on successively more information about f. Hope: 9t such that p t ;p t+1 ;::: each do a good enough job computing f. Hope realized? Depends on M, f and ones exact criterion of success. Next: criteria of success, then examples. 3

4 f(0);f(1);::: Criteria of Success In,! M,! Out p 0 ;p 1 ;:::;j p t ;::: Suppose a 2 N [ fg. a for anomaly count. For a =, a stands for nitely many. Suppose F R 0;1, the class of all (total) computable functions : N! f0; 1g. F 2 Ex a def, (9M)(8f 2 F) [M fed f(0);f(1);:::;outputs p 0 ;p 1 ;::: ^ (9t)[p t = p t+1 = ^ p t computes f except at up to a inputs ]]: F 2 Bc a def, (9M)(8f 2 F) [M fed f(0);f(1);:::;outputs p 0 ;p 1 ;::: ^ (9t)[p t ;p t+1 ;::: each computes f except at up to a inputs]]: 4

5 Examples For k 1, P k def = class all 0-1 val. funs. comp. by multi-tape TMs in O(n k ) time, w/ n, length input. P def = S P k. Q k def = class all 0-1 val. funs. comp. in O(n k (log n) 2 ) time. P k Q k P k+1. ( Gold'67) P 2 Ex 0. P k 2 Ex 0 too (w/ each output conj. running in k- deg. poly time). CF, class all char. funs. of context free langs, 2 Ex 0. (Case & Smith'78 + :::) Ex 0 Ex 1 Ex 2 Ex Bc 0 Bc 1 Bc. (Harrington'83) R 0;1 2 Bc. Later: R 0;1 2 Bc witnessed by some M outputting only total, 0-1 valued conjectures. 5

6 Basic Notation ( 1 8 x) means for all but nitely many x 2 N. U def = ff 2 R 0;1 j ( 1 8 x)[f(x) = 1]g ( P 1 ). def p = the partial computable function : N! N computed by Turing machine program (number) p. ' TM = the runtime of Turing machine program (number) p on input x, if p halts on x, and undened, otherwise. TM p WS p (x) def = the work space used by Turing machine program (number) p on input x, if p halts on x, and undened, otherwise. (x) def U REG, class all char. funs. of reg. langs. 6

7 Complexity Deciency f[n] def = the sequence f(0);:::;f(n, 1). M(f[n]) def = M's output based only on f[n]. Can suppose without loss of generality M(f[n]) is always dened. Proposition 9M witnessing REG 2 Ex 0 such that (8n)[ TM M(f[n]) jxj + 1 ^ WS M(f[n]) 0]. By contrast: Theorem (improves Sipser) Suppose k 1 and that M witnesses that Q k 2 (Ex [ Bc 0 ). Then: (8k-degree polynomials p)(9f 2 U ) ( 1 8 n)( 1 8 x)[ TM M(f[n]) (x) > p(jxj)]. Up gen. of M from P k to Q k entails run-times of M's successful outputs on some f 2 U worse than any preassigned k-deg poly bnd. Conj: 1 log n fac. 7! arb. slow-grow. P 1 -fun. 7

8 What about Bc m for m > 0? ( 1 9 x) means exists innitely many x 2 N such that.... Theorem 9M outputting only total, polynomial time conjectures and witnessing both P 2 Bc 1 and P 1 2 Ex 0 such that (8f 2 P 1 )(9 linear polynomial p) ( 1 9 x)[ TM M(f) (x) p(jxj)]. Tentative Theorem Suppose k; m > 0 and that M witnesses that Q k 2 Bc m. Then: (8k-degree polynomials p)(9f 2 U ) ( 1 8 n)( 1 9 x)[ TM M(f[n]) (x) > p(jxj)]. 8

9 Information Deciency M(f) denotes M's nal output program, if any. The next two results together nicely contrast with previous 2 slides. Recall: their theorems featured criteria Ex ; Bc m. Theorem 9M outputting only total poly time conjs. & witnessing both P 2 Bc & P 1 2 Ex 0 such that (8f 2 P 1 )[ TM M(f) 2 O(jxj)]. Theorem Suppose T is any true, computably axiomatized extension of rst order PA (hence, T is a safe, algorithmic extractor of information). Suppose k 1 and M witnesses that Q k 2 Bc. Then: (9f 2 U )( 1 8 n)[t 6` ' TM M(f[n]) is computable in O(jxjk ) time ]. While the learned programs for functions in U can have excellent run times, some of these learned programs will be informationally decient since we cannot prove from them even considerably weaker upper bounds on the run times of any total nite variants of the functions they compute. 9

10 How Are These Results Proved? Positive results are proved by careful programming with tricks from: J. Royer and J. Case, Subrecursive Programming Systems: Complexity and Succinctness, Research monograph in Progress in Theoretical Computer Science, Birkhauser Boston, The rst two negative results above require tricks from the monograph, and, for Bc m or ( 1 8 x), aggressive diagonalization. The other/last negative result above depends on (extensions of) delicate inseparability results also from the same monograph. The rst negative result but only for Ex and with (9x) for ( 1 8 x) can also be done with such inseparabilities. Ditto re use of inseparabilities for the remaining negative results on further slides (and other results too). 10

11 2-Subsets of N 2 sets ( N) are (by denition) those dened by some predicate of the form (9u)(8v)R, where R is some computable numerical predicate. (Post, Gold, Putnam, Shapiro) 2 sets are characterized as the lim-r.e. sets, i.e., as those sets S which can be accepted by some mind-changing algorithm, where, for x 2 S, on x outputs an innite sequence of 0's and 1's eventually ending in all 1's, and, for x =2 S, on x outputs an innite sequence of 0's and 1's not eventually ending in all 1's. 0 = NO; 1 = YES. 11

12 2-Inseparability S A B For above N, S separates B from A. B is 2 -inseparable from A def, A and B are disjoint, but no 2 set S separates B from A. I.e., it's algorithmically impossible in the limit to tell elements of B from those of A an algorithmic deciency. Extension of Monograph: if progr. sys. for Q k, fi j i 2 (Q k, P k )g 2 -insep. from fi j i 2 U g. If for CF, fi j i 2 (CF, REG)g 2 -insep. from fi j i 2 U g. Monograph: such insep. characterize corr. program succinctness: general systems have shorter programs for some f 2 U than less general systems. 12

13 Inseparable Classes of Functions S A B In above picture A; B; S are classes of functions : N! N with S separating B from A. (fn) 2 classes of functions are (by denition) those dened by some predicate of the form (9u)(8v)R, where R is some computable predicate of a function parameter too where the function parameter is treated like an oracle to be algorithmically quizzed as to its values. B is (fn) 2 -inseparable from A def, A and B are disjoint, but no (fn) 2 class S separates B from A. I.e. it's algorithmically impossible in the limit to tell elements of B from those of A another algorithmic deciency. 13

14 Needed Corollary and Thoughts Corollary Suppose k 1. Then: (Q k,p k ) and (CF,REG) are each (fn) 2 -inseparable from U. FV(B) def = fpartial functions : N! N j is a nite variant of some f 2 Bg. In the following, think of: { A as a very modest class, e.g., U ; { B as a more immodest class, e.g., (Q k, P k ) or (CF, REG); and { G as a set of nice programs, but not for any elements of FV(B). Lesson: in the present context, the deciences re algorithmicity, complexity, and size or information are birds of a feather. 14

15 More Information Deciency Theorem Suppose that (i-iii). (i) B is (fn) 2 -inseparable from A. (ii) G is an r.e. set of TM-programs so that (FV(B) \ f' TM p j p 2 Gg) = ;. (iii) M witnesses that B 2 Bc. Then: (9f 2 A)( 1 8 n)[m(f[n]) =2 G]. For r.e. G based on computable axiomatizations, we get our last negative provability result above re Q k vs. U AND the following. Theorem Suppose T is any true, computably axiomatized extension of rst order PA (hence, T is a safe, algorithmic extractor of information). Suppose M witnesses that CF 2 Bc. Then: (9f 2 U )( 1 8 n) [T 6` ' TM M(f[n]) decides a regular language ]. 15

16 More Complexity Deciency Theorem Suppose that (i-iii). (i) B is (fn) 2 -inseparable from A. (ii) G is the complement of an r.e. set of TMprograms so that (FV(B)\f' TM p j p 2 Gg) = ;. (iii) M witnesses (A [ B) 2 Ex. Then: (9f 2 A)[M(f) =2 G]. Theorem Suppose k 1 and M that CF 2 Ex. Then: (9f 2 U )(9x)[ WS M(f) (x) k]. witnesses Don't know if (9x) just above can be changed to ( 1 9 x) or ( 1 8 x). 16

17 Conclusions For Ex ; Bc m, upping generality from, e.g., learning P k to Q k, forces learned progs. for some f 2 U to run worse than any preassigned k-deg poly time bound. For Bc, up generality from, e.g., learning P k to Q k, can get linear time learned progs. for P 1, but cannot safely, alg. prove from learned programs for some f 2 U good run times for tot. nite variants. Algorithmic deciency in limit telling (Q k, P k ) or (CF, REG) from U entails exists complexity/information deciency of learned progs. Su. conds. yield above &: for M witnessing CF 2 Bc, can't prove, for an f 2 U, that M's learned progs. for f decide regular languages; and, for M witnessing CF 2 Ex & k 1, for some f 2 U, M's learned program on f, uses WS k. 17

Generality s Price: Inescapable Deficiencies in Machine-Learned Programs USA.

Generality s Price: Inescapable Deficiencies in Machine-Learned Programs John Case 1, K-J. Chen 2, Sanjay Jain 3, Wolfgang Merkle 4, and James S. Royer 5 1 Dept. of Computer and Information Sciences, University