Notes on Information, Reference, and Machine Thirdness

Transcription

1 Notes on Information, Reference, and Machine Thirdness Rafael Alvarado Data Science Institute Human & Machine Intelligence Group Humanities Informatics Lab 24 July 2018

2 Overview Paul proposes that neural networks (ANNs) may be characterized by their capacity to preserve referential content. Where refence and representation are to be understood to be independent of human cognition. I propose that to explore this idea entails a broad project to combine semiotics and information theory in order to provide a suitable theory of reference. Since reference is an element of semiotics, and we want to measure referential content preservation.

3 Overview Specifically, we want to use information theory to develop a non-semantic theory of reference. This project appears doomed, since information theory (Shannon 19848) begins by separating semantics from information, and reference is central to semantics. Reference and information ought to have nothing to do with each other. However, this limitation may be more apparent than real. It follows from our confused usage of the term information, and an ossified understanding of reference.

4 Four Meanings of Information 1. Instruction Both verb and noun. Rooted in metaphysics of form and matter. Roman and Medieval usage. Advice, education. 2. News Both Austen information (private) and content of newspapers (public). Particular, actionable, time-sensitive, remote knowledge. 3. Entropy See Shannon (1948). A measurable property of messages. Bell Labs solution to engineering problem with telegraphs, etc. 4. Form See Bateson (1970) Form, Substance, and Difference. Information travels from the wood and paper into my retina. It then gets picked up and worked on by this fancy piece of computing machinery in my head. Associated with the English school of information theory.

5 The four meanings are related both historically and conceptually. For our purposes, we want to focus on information as entropy but not lose its connection to information as form. Even given the narrow definition of Shannon, the grammar of the word is confused.

6 Usages of Shannon Information 1. As entropy: The more information, the less determinate the output of a random process (i.e. more choices for prediction). Information as uncertainty. 2. As reduction of entropy: Implied by information gain. The more information we gain, the more determinate the random process (i.e. fewer choices). Information as certainty. 3. As the bit, the elementary unit of information: No reference at all to entropy or certainty. Information is just the sum of bits in a container (e.g. a hard drive or newspaper). Effectively refers to the space in a sequence that can hold a bit. Entropy measured in bits, therefore information is prior to entropy.

7 The mathematical formulae used to define the bit, entropy, relative entropy, and mutual information (Cover 2006), are coherent, indeed elegant, but the language we use to describe the phenomena they treat is incoherent. Distance, time, and mass do not have these issues. We are very clear about their magnitudes and units. More is more, less is less.

8 Proposed Definitions Bit A member of the smallest possible alphabet (e.g. 0 and 1). Occupies a single space in a sequence. Not a number of per se, but can be treated as one. Sequences of bits are messages. Entropy The irreducible complexity of a random process (Cover). Measured in bits. Inferred through, but not identical to, proxies, such as optimal length of output messages. Information Proportional to the inverse of entropy. Thus, more information means more certainty about a process.

9 Benefits of this Definition of Information Information is not simply a variant way of representing entropy; the inverse of entropy is an index of form and pattern, which is interestingly related to the concept of complexity (Cover and Thomas 2006 [1990]: 1). Sequence is a kind of pattern that lends itself to analysis. The relationship of pattern to complexity remains to be explicated. Also, in this definition, information flows throughout a system. Finally, allows us to say things like a well-trained neural network is literally a well-informed neural network. We are back to learning as informing...

10 Conjectures about Information Shannon information is a property of messages. The outputs of random processes are always represented as messages, which are sequences of (so-called) symbols drawn from a so-called alphabet, even when these outputs manifest as continuous signals from non-human sources. Information is physical but not material. It can be measured, but has neither mass nor energy. See Wiener s principle: Information is information, not matter or energy. No materialism which does not admit this can survive at the present day. (1948: 132) Inescapable hylomorphism!

11 thirdness C A firstness B secondness

12 Reference A stands for B A: vehicle, B: target stands for means 1 refers to denotes is a (name sign symbol ) of Always implicitly to someone (a mind) A stands for B to C C makes (and stores) the connection between A and B Peirce s thirdness Always asymmetric A B B A A is a substitute for B

13 Conjectures about Asymmetry Asymmetry is hierarchical. A sign points from the plane of signs to another plane of referents, which may be images or other signs. Asymmetry is a requirement to make inferences about B from A. We make inferences about B by manipulating A. Signs are used to model the relationships between things. Asymmetry is required to communicate about B in its absence. Signs are required to have thoughts about things that are not present. Referential relationships collectively form a directed graph of references. That is, references participate in semantic graphs.

14 The asymmetry of reference implies thirdness. Firstness and secondness are sufficient to describe mere associations, but thirdness implies a plane of inference in which signs can be connected into propositional networks comprising analytic and synthetic statements. IOW, reference implies inference.

15 This characterization implies a distinction between deictic and logical reference. Deictic reference involves signs from the plane of inference pointing to entities outside of this plane (so-called things ). Logical reference involves signs within the plane pointing to each other, by means of propositional forms ( A is B ) to form chains of inference.

16 Thirdness Most theorists of reference agree that the referential relationship is always constructed by means of thirdness. Ogden and Richards called this thought. Saussure conceived of this as a system of differences. Frege s Sinn, translated as Sense, occupies this space. Connotations, which derive from the association of a word with other words, may be said to occupy this space as well. Kant s transcendental unity of apperception goes here. In fact, thirdness is something of a black box whose contents are characterized by various theorists.

17 Note that there is room in the concept of thirdness to accommodate variability in planes of inference, i.e. modes of rationality such as logical, so-called pre-logical, nonconscious, etc.

18 A Non-Semantic Theory of Reference First, eliminate requirement of asymmetry. Reference without asymmetry is association. Associations are of two kinds: by contiguity or similarity. Hume posits causality, but this is actually a kind of contiguity (in time). Then, recover thirdness through application of information theory to associations. An a posteriori analytic approach to reference. Reference is in fact built on top of association: these are motivations for reference, which is always conventional (thirdness)

19 Machine Thirdness We know that associations may be generated mechanically through the application of information theory. Contiguity informed by means of mutual information and association rules. Similarity informed by means of various distance measures (e.g. Euclidean, cosine, Jensen- Shannon divergence, Jaccard index, etc. See Tiel and Latour, 1995, The Hume Machine: Can Association Networks Do More Than Formal Rules? But what does mechanical thirdness look like? Do machines that learn contain anything like a plane of inference?

20 Information and Thirdness The question is, what is the relationship between thirdness and information? How does our definition of information help us to answer this question? Complexity? Look at work on opening the black box...

21 Neural Networks We know that ANNs trade model interpretability for predictive accuracy. Model interpretability is required to perform inference. It is important to note, however, that ANNs do generate models they are just enormously complex. In principle, these models form an inferential graph that occupies the place of thirdness in establishing the relationship between input variables (A) and output labels (B). Recently, an understanding of the structure of this black box has begun to emerge.

22 Neural Networks 1. Features and Labels 1. Predictor variables stand for labels. 2. Machines model this relationship through training. 2. Information Bottleneck (Tishby 2000) 1. First layers do memorization and basic recognition. 2. Last layers do compression and optimization. 3. Machine learns (is informed) by means of throwing out extraneous data not a simple mirroring as Hume would have it. 3. Transfer Learning 1. First layers discover common patterns. 2. Last layers are fine-tuning. 3. First layers can be inherited and locked (frozen).

23 Shwartz-Ziv, Ravid, and Naftali Tishby Opening the Black Box of Deep Neural Networks via Information. ArXiv: [Cs], March.

24 Geoffrey Hinton, a pioneer of deep learning who works at Google and the University of Toronto, ed Tishby after watching his Berlin talk. It s extremely interesting, Hinton wrote. I have to listen to it another 10,000 times to really understand it, but it s very rare nowadays to hear a talk with a really original idea in it that may be the answer to a really major puzzle. Wolchover, ByNatalie. n.d. New Theory Cracks Open the Black Box of Deep Learning. Quanta Magazine. Accessed July 24,

25 End