Mathematics of the Information Age
|
|
- Charla Phillips
- 6 years ago
- Views:
Transcription
1 Mathematics of the Information Age
2 Material ages The Stone Age From - to about 4000BC The Bronze Age From 2300 BC to 500 BC The Iron Age From 800 BC to 100 AD
3 The Information Age Begins in 1948 with the work of Claude Shannon at Bell Labs
4 What do the codes used for sending messages back from spacecraft have in common with genes on a molecule of DNA? How is it that the second law of thermodynamics, a physicist s discovery, is related to communication? Why are the knotty problems in the mathematical theory of probability connected with the way we express ourselves in speech and communication? The answer to all of these questions is information Jeremy Campbell, Grammatical Man,1982
5 I shall argue that this information flow, not energy per se, is the prime mover of life that molecular information flowing in circles brings forth the organization we call organism and maintains it against the ever present disorganizing pressures in the physics universe. So viewed, the information circle becomes the unit of life. Werner Lowenstein, The Touchstone of Life, 2000
6 Aspects of Information?
7 Practical Perceptual Physical All have something to do with communication
8 Aspects of information the theory The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated to some system with certain physical or contextual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. Claude Shannon, The Mathematical Theory of Communication, 1948
9 Prior condition for communication to be possible: The sender and receiver both have to have the same set of all possible messages, or be able to construct it. They need the same codebook
10 The most famous codebook in history?
11
12 How do we measure information? (In Shannon s theory, Information becomes quantitative.)
13 Remember Shannon s quote: The significant aspect is that the actual message is one selected from a set of possible messages. How to quantify the process of selection?
14 Let s play 20 questions! I m thinking of a famous person. (But remember, we both know all the famous people.)
15 1. The person is Brad Osgood
16 1. The person is Brad Osgood 2. The person is Rebecca Osgood
17 1. The person is Brad Osgood 2. The person is Rebecca Osgood 3. The person is Miles Osgood 4. The person is Madeleine Osgood
18 1. The person is Brad Osgood 2. The person is Rebecca Osgood 3. The person is Miles Osgood 4. The person is Madeleine Osgood 5. The person is Ruth Osgood 6. The person is Herbert Osgood 7. The person is Lynn Osgood 8. The person is Alex Beasley 9. The person is Thomas Faxon 10. The person is Virginia Faxon 11. The person is Thomas Faxon, Jr. 12. The person is Meer Deiters 13. The person is Francisca Faxon 14. The person is Pia Faxon 15. The person is George W. Bush 16. The person is Saddam Hussein
19 Brad says: Who needs 20 questions. I bet I can pick out any object (in English) by asking 18 questions. OK, maybe 19. Hah! What is the basis for this bold claim? Is it justified? In the real version of 20 questions the sender says that object is animal, mineral or vegetable to allow the receiver to narrow down their questions. Just how many things can you determine by asking 20 questions?
20 2 18 = 261, = 524,288 The number of entries in the 1989 edition of the Oxford English Dictionary is 291, = 1,048,576
21 Impress your friends. I can pick any name out of the Stanford phone book in N questions
22 The unit of information is the bit How many bits how many yes-no questions are needed to select one particular message from a set of possible messages? The possible messages are encoded into sequences of bits. In practice, 0 s and 1 s (off, on; no, yes). Many coding schemes are possible, some more efficient or reliable than others. There are many ways to play 20 questions
23 General definition of amount of information Suppose there are N possible messages. The amount of information in any particular message is I = log 2 N (unit is bits) (Same thing as saying 2 I =N) What does it mean to say that the amount of information in a message is, e.g., 3.45 bits?
24 I m more famous than you are In any practical application not all messages are equally probable. How can we measure information taking probabilities into account?
25 1. The person is Brad Osgood 2. The person is Brad Osgood 3. The person is Brad Osgood 4. The person is Brad Osgood 5. The person is George W. Bush 6. The person is Saddam Hussein 7. The person is Colin Powell 8. The person is Condoleezza Rice Playing the game many times, how many questions do you think you d need, on average to pick out a particular message?
26 Is the person in the group 1 through 4? Yes. No One question resolves the uncertainty. Need two more questions, for a total of three. Brad Osgood occurs 4 out of 8 times: Probability 4/8=1/2 I( Brad Osgood ) = 1 Everybody else occurs 1 out of 8 times: Probability 1/8 I( George W. ) = 3
27 In general, if a message S occurs with probability p then I(S) = log 2 (1/p) If we have N messages (the source ) S 1, S 2,,S N occurring with probabilities p 1,p 2,,p N then the average information of the source as a whole (the entropy of the source ) is the weighted average of the information of the individual messages: H=p 1 log 2 (1/p 1 )+p 2 log 2 (1/p 2 )+ + p N log 2 (1/p N )
28 Can you improve your estimate on how many questions it should take to pick a name out of the Stanford phone book?
29 Shannon defined entropy as a measure of average information in a source (the collection of possible messages), taking probabilities into account. H=p 1 log 2 (1/p 1 )+p 2 log 2 (1/p 2 )+ + p N log 2 (1/p N ) And he proved:
30 Noiseless Source Coding Theorem: For any coding scheme the average length of a codeword is at least the entropy. This gives a lower bound to our cleverness
31 Shannon defined the capacity of a channel as a measure of how much information it could transmit. And he proved:
32 Channel Coding Theorem: A channel with capacity C is capable, with suitable coding, of transmitting at any rate less than C bits per symbol with vanishingly small probability of error. For rates greater than C the probability of error cannot be made arbitrarily small.
33 Most great physical and mathematical discoveries seem trivial after you understand them. You say to yourself: I could have done that. But as I hold the tattered journal containing Claude Shannon s classic 1948 paper A Mathematical Theory of Communication I see yellowed pages filled with vacuum tubes and mechanisms of yesteryear, and I know I could never have conceived the insightful theory of information shining through these glossy pages of archaic font. I know of no greater work of genius in the annals of technological thought. Robert W. Lucky, Silicon Dreams, 1989
34 The course syllabus
35 Analog Signal (e.g. Music, Speech, Images) A to D converte Digitized signal (0s and 1 s) Compression (e.g. MP3) Add error correction (e.g fixes scratches in CDs) Noise! The Channel (e.g. Fiber optics, the Internet, Computer memory) Correct errors (Remove redundancy) Uncompress D to A converte
36 It took awhile for the technology to catch up with Shannon s theory
37 The news from Troy In Agamemnon by Aeschylus The fall of Troy was signaled by a beacon. The play opens with a watchman who waited for 12 years for a single piece of news: the promised sign, the beacon flare to speak from Troy and utter one word, `Victory!'."
38 The news from Gondor
39 The news from Paris A message was spelled out, symbol by symbol, and relayed from one station to the next. Operators at intermediate stations were allowed to know only portions of the codebook. The full codebook, which had over 25,000 entries, was given only to the inspectors.
40 The network 1820 s 1850 s
41 High Tech of the mid 19 th Century 1824 Samuel F.B. Morse, an art instructor, learns about electromagnetism 1831 Joseph Henry demonstrates an electromagnetic telegraph with a one mile run in Albany, New York Morse demonstrates his electric telegraph in New York 1837 Wheatstone and Cooke set up British electric telegraph. Transatlantic cables around 1904
42 The first shot in the second William Thomson (later Lord Kelvin, ) On the theory of the electric telegraph, Proceedings of the Royal Society, 1855 industrial revolution Answered the question of why signals smear out over a long cable.
43 Communication became mathematical! Surely this must have been hailed as a breakthrough!
44 I believe nature knows no such application of this law and I can only regard it as a fiction of the schools; a forced and violent application of a principle in Physics, good and true under other circumstances, but misapplied here. Edward Whitehouse, chief electrician for the Atlantic Telegraph Company, speaking in 1856.
45 Right. The first transatlantic cable used Whitehouse s specifications, not Thomson s The continents were joined August 5, 1858 (after four previous failed attempts). The first successful message was sent August 16. The cable failed three weeks later. Whitehouse insisted on using high voltage, disregarding Thomson s analysis
46 The rise of electrical networks telegraph, telephone and beyond
47 Broadway & John Street, New York 1890
48 Gerard Exchange, London, 1926
49 What s wrong with this picture?
50 Wireless Guglielmo Marconi ( )
51 The last of the great data networks?
52
53 First need a mathematical description of signals What kinds of signals? Speech Music Images All can be described via Fourier analysis
54 Major Secret of the Universe Every signal has a spectrum
An introduction to basic information theory. Hampus Wessman
An introduction to basic information theory Hampus Wessman Abstract We give a short and simple introduction to basic information theory, by stripping away all the non-essentials. Theoretical bounds on
More informationPart I. Entropy. Information Theory and Networks. Section 1. Entropy: definitions. Lecture 5: Entropy
and Networks Lecture 5: Matthew Roughan http://www.maths.adelaide.edu.au/matthew.roughan/ Lecture_notes/InformationTheory/ Part I School of Mathematical Sciences, University
More informationThe Liar Game. Mark Wildon
The Liar Game Mark Wildon Guessing Games Ask a friend to thinks of a number between 0 and 15. How many NO/YES questions do you need to ask to find out the secret number? Guessing Games Ask a friend to
More informationA Mathematical Theory of Communication
A Mathematical Theory of Communication Ben Eggers Abstract This paper defines information-theoretic entropy and proves some elementary results about it. Notably, we prove that given a few basic assumptions
More information(Classical) Information Theory III: Noisy channel coding
(Classical) Information Theory III: Noisy channel coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract What is the best possible way
More information3F1 Information Theory, Lecture 1
3F1 Information Theory, Lecture 1 Jossy Sayir Department of Engineering Michaelmas 2013, 22 November 2013 Organisation History Entropy Mutual Information 2 / 18 Course Organisation 4 lectures Course material:
More informationlog 2 N I m m log 2 N + 1 m.
SOPHOMORE COLLEGE MATHEMATICS OF THE INFORMATION AGE SHANNON S THEOREMS Let s recall the fundamental notions of information and entropy. To repeat, Shannon s emphasis is on selecting a given message from
More information17.1 Binary Codes Normal numbers we use are in base 10, which are called decimal numbers. Each digit can be 10 possible numbers: 0, 1, 2, 9.
( c ) E p s t e i n, C a r t e r, B o l l i n g e r, A u r i s p a C h a p t e r 17: I n f o r m a t i o n S c i e n c e P a g e 1 CHAPTER 17: Information Science 17.1 Binary Codes Normal numbers we use
More informationUNIT I INFORMATION THEORY. I k log 2
UNIT I INFORMATION THEORY Claude Shannon 1916-2001 Creator of Information Theory, lays the foundation for implementing logic in digital circuits as part of his Masters Thesis! (1939) and published a paper
More informationMassachusetts Institute of Technology. Problem Set 4 Solutions
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Department of Mechanical Engineering.050J/2.110J Information and Entropy Spring 2003 Problem Set 4 Solutions
More informationCommunication Engineering Prof. Surendra Prasad Department of Electrical Engineering Indian Institute of Technology, Delhi
Communication Engineering Prof. Surendra Prasad Department of Electrical Engineering Indian Institute of Technology, Delhi Lecture - 41 Pulse Code Modulation (PCM) So, if you remember we have been talking
More informationInformation in Biology
Information in Biology CRI - Centre de Recherches Interdisciplinaires, Paris May 2012 Information processing is an essential part of Life. Thinking about it in quantitative terms may is useful. 1 Living
More informationchannel of communication noise Each codeword has length 2, and all digits are either 0 or 1. Such codes are called Binary Codes.
5 Binary Codes You have already seen how check digits for bar codes (in Unit 3) and ISBN numbers (Unit 4) are used to detect errors. Here you will look at codes relevant for data transmission, for example,
More informationA Room-Sized Computer in Your Digital Music Player By ReadWorks
By ReadWorks It was the morning of July 7, 1941, and problems were mounting for General George Marshall. American Army code breakers had intercepted and deciphered conversations between Japanese leaders
More information( c ) E p s t e i n, C a r t e r a n d B o l l i n g e r C h a p t e r 1 7 : I n f o r m a t i o n S c i e n c e P a g e 1
( c ) E p s t e i n, C a r t e r a n d B o l l i n g e r 2 0 1 6 C h a p t e r 1 7 : I n f o r m a t i o n S c i e n c e P a g e 1 CHAPTER 17: Information Science In this chapter, we learn how data can
More informationWelcome to Comp 411! 2) Course Objectives. 1) Course Mechanics. 3) Information. I thought this course was called Computer Organization
Welcome to Comp 4! I thought this course was called Computer Organization David Macaulay ) Course Mechanics 2) Course Objectives 3) Information L - Introduction Meet the Crew Lectures: Leonard McMillan
More information9 THEORY OF CODES. 9.0 Introduction. 9.1 Noise
9 THEORY OF CODES Chapter 9 Theory of Codes After studying this chapter you should understand what is meant by noise, error detection and correction; be able to find and use the Hamming distance for a
More informationPolynomial Codes over Certain Finite Fields
Polynomial Codes over Certain Finite Fields A paper by: Irving Reed and Gustave Solomon presented by Kim Hamilton March 31, 2000 Significance of this paper: Introduced ideas that form the core of current
More information6.02 Fall 2012 Lecture #1
6.02 Fall 2012 Lecture #1 Digital vs. analog communication The birth of modern digital communication Information and entropy Codes, Huffman coding 6.02 Fall 2012 Lecture 1, Slide #1 6.02 Fall 2012 Lecture
More informationLMS Popular Lectures. Codes. Peter J. Cameron
LMS Popular Lectures Codes Peter J. Cameron p.j.cameron@qmul.ac.uk June/July 2001 Think of a number... Think of a number between 0 and 15. Now answer the following questions. You are allowed to lie once.
More informationCommunication Engineering Prof. Surendra Prasad Department of Electrical Engineering Indian Institute of Technology, Delhi
Communication Engineering Prof. Surendra Prasad Department of Electrical Engineering Indian Institute of Technology, Delhi Lecture - 3 Brief Review of Signals and Systems My subject for today s discussion
More informationEntropies & Information Theory
Entropies & Information Theory LECTURE I Nilanjana Datta University of Cambridge,U.K. See lecture notes on: http://www.qi.damtp.cam.ac.uk/node/223 Quantum Information Theory Born out of Classical Information
More informationCSCI 2570 Introduction to Nanocomputing
CSCI 2570 Introduction to Nanocomputing Information Theory John E Savage What is Information Theory Introduced by Claude Shannon. See Wikipedia Two foci: a) data compression and b) reliable communication
More informationInformation Theory (Information Theory by J. V. Stone, 2015)
Information Theory (Information Theory by J. V. Stone, 2015) Claude Shannon (1916 2001) Shannon, C. (1948). A mathematical theory of communication. Bell System Technical Journal, 27:379 423. A mathematical
More informationMP203 Statistical and Thermal Physics. Jon-Ivar Skullerud and James Smith
MP203 Statistical and Thermal Physics Jon-Ivar Skullerud and James Smith October 3, 2017 1 Contents 1 Introduction 3 1.1 Temperature and thermal equilibrium.................... 4 1.1.1 The zeroth law of
More informationQuantum Teleportation Pt. 1
Quantum Teleportation Pt. 1 PHYS 500 - Southern Illinois University April 17, 2018 PHYS 500 - Southern Illinois University Quantum Teleportation Pt. 1 April 17, 2018 1 / 13 Types of Communication In the
More informationPhysical Systems. Chapter 11
Chapter 11 Physical Systems Until now we have ignored most aspects of physical systems by dealing only with abstract ideas such as information. Although we assumed that each bit stored or transmitted was
More informationD CUME N TAT ION C 0 R P 0 R A T E D 2,21 CONNECTICUT AVENUE, N. W,
44 D CUME N TAT ION C 0 R P 0 R A T E D 2,21 CONNECTICUT AVENUE, N. W, rs WlLISTS IN BASIC AND APPLIED INFORMATION THEORY WASHINGTON 8, D. C. C 0 L U M B I A 5-4 5 7 7 SOMMUNICATION THEORY AND STORAGE
More informationInformation in Biology
Lecture 3: Information in Biology Tsvi Tlusty, tsvi@unist.ac.kr Living information is carried by molecular channels Living systems I. Self-replicating information processors Environment II. III. Evolve
More informationAn Introduction to Information Theory: Notes
An Introduction to Information Theory: Notes Jon Shlens jonshlens@ucsd.edu 03 February 003 Preliminaries. Goals. Define basic set-u of information theory. Derive why entroy is the measure of information
More informationDigital Communications III (ECE 154C) Introduction to Coding and Information Theory
Digital Communications III (ECE 154C) Introduction to Coding and Information Theory Tara Javidi These lecture notes were originally developed by late Prof. J. K. Wolf. UC San Diego Spring 2014 1 / 8 I
More informationDefinition of geometric vectors
Roberto s Notes on Linear Algebra Chapter 1: Geometric vectors Section 2 of geometric vectors What you need to know already: The general aims behind the concept of a vector. What you can learn here: The
More informationAn Introduction to (Network) Coding Theory
An to (Network) Anna-Lena Horlemann-Trautmann University of St. Gallen, Switzerland April 24th, 2018 Outline 1 Reed-Solomon Codes 2 Network Gabidulin Codes 3 Summary and Outlook A little bit of history
More informationThe Laws of Thermodynamics and Information and Economics
The Laws of Thermodynamics and Information and Economics By: William Antonio Boyle, PhD Prince George s Community College Largo, Maryland, USA 17 October 2017 Thermodynamics Began as the study of steam
More informationShannon s Noisy-Channel Coding Theorem
Shannon s Noisy-Channel Coding Theorem Lucas Slot Sebastian Zur February 2015 Abstract In information theory, Shannon s Noisy-Channel Coding Theorem states that it is possible to communicate over a noisy
More informationShannon's Theory of Communication
Shannon's Theory of Communication An operational introduction 5 September 2014, Introduction to Information Systems Giovanni Sileno g.sileno@uva.nl Leibniz Center for Law University of Amsterdam Fundamental
More informationIntroduction to Algebra: The First Week
Introduction to Algebra: The First Week Background: According to the thermostat on the wall, the temperature in the classroom right now is 72 degrees Fahrenheit. I want to write to my friend in Europe,
More information! Where are we on course map? ! What we did in lab last week. " How it relates to this week. ! Compression. " What is it, examples, classifications
Lecture #3 Compression! Where are we on course map?! What we did in lab last week " How it relates to this week! Compression " What is it, examples, classifications " Probability based compression # Huffman
More informationPlease bring the task to your first physics lesson and hand it to the teacher.
Pre-enrolment task for 2014 entry Physics Why do I need to complete a pre-enrolment task? This bridging pack serves a number of purposes. It gives you practice in some of the important skills you will
More informationITCT Lecture IV.3: Markov Processes and Sources with Memory
ITCT Lecture IV.3: Markov Processes and Sources with Memory 4. Markov Processes Thus far, we have been occupied with memoryless sources and channels. We must now turn our attention to sources with memory.
More informationto mere bit flips) may affect the transmission.
5 VII. QUANTUM INFORMATION THEORY to mere bit flips) may affect the transmission. A. Introduction B. A few bits of classical information theory Information theory has developed over the past five or six
More informationInt er net Saf et y Tip s
BE CAREFUL AS: Facebook oft en means People oft en pret end t o be people t hey are not so be wary of t his!! Int er net Saf et y Tip s N ever accept people you do not know. Never give out your real name
More information6.02 Fall 2011 Lecture #9
6.02 Fall 2011 Lecture #9 Claude E. Shannon Mutual information Channel capacity Transmission at rates up to channel capacity, and with asymptotically zero error 6.02 Fall 2011 Lecture 9, Slide #1 First
More information3F1 Information Theory, Lecture 3
3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2011, 28 November 2011 Memoryless Sources Arithmetic Coding Sources with Memory 2 / 19 Summary of last lecture Prefix-free
More informationIntroducing Inspector Tippington
Introducing Inspector Tippington Inspector Tippington is a world-famous detective who is retired from Scotland Yard. He is also an expert in world history. He has spent his life traveling around the world
More informationCHAPTER 7: TECHNIQUES OF INTEGRATION
CHAPTER 7: TECHNIQUES OF INTEGRATION DAVID GLICKENSTEIN. Introduction This semester we will be looking deep into the recesses of calculus. Some of the main topics will be: Integration: we will learn how
More information3F1 Information Theory, Lecture 3
3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2013, 29 November 2013 Memoryless Sources Arithmetic Coding Sources with Memory Markov Example 2 / 21 Encoding the output
More information1. Study the following Vocabulary Words to be defined: Prehistory, History, Geography, 5 Themes of Geography, Legacy
Social Studies Mr. Poirier Introduction Test - Study Guide Study Guide given in class on Monday September 18, 2017 Introduction Unit Test - Thursday September 21, 2017 1. Study the following Vocabulary
More informationMath 138: Introduction to solving systems of equations with matrices. The Concept of Balance for Systems of Equations
Math 138: Introduction to solving systems of equations with matrices. Pedagogy focus: Concept of equation balance, integer arithmetic, quadratic equations. The Concept of Balance for Systems of Equations
More informationLecture 11: Information theory THURSDAY, FEBRUARY 21, 2019
Lecture 11: Information theory DANIEL WELLER THURSDAY, FEBRUARY 21, 2019 Agenda Information and probability Entropy and coding Mutual information and capacity Both images contain the same fraction of black
More informationPERFECT SECRECY AND ADVERSARIAL INDISTINGUISHABILITY
PERFECT SECRECY AND ADVERSARIAL INDISTINGUISHABILITY BURTON ROSENBERG UNIVERSITY OF MIAMI Contents 1. Perfect Secrecy 1 1.1. A Perfectly Secret Cipher 2 1.2. Odds Ratio and Bias 3 1.3. Conditions for Perfect
More informationMassachusetts Institute of Technology
Name (1%): Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Department of Mechanical Engineering 6.050J/2.110J Information and Entropy Spring 2006 Issued:
More informationWe saw last time how the development of accurate clocks in the 18 th and 19 th centuries transformed human cultures over the world.
We saw last time how the development of accurate clocks in the 18 th and 19 th centuries transformed human cultures over the world. They also allowed for the precise physical measurements of time needed
More informationLesson 32. The Grain of Wheat. John 12:20-26
L i f e o f C h r i s t from the gospel of J o h n Lesson 32 The Grain of Wheat John 12:20-26 Mission Arlington Mission Metroplex Curriculum 2010 Created for use with young, unchurched learners Adaptable
More informationInformation Theory and Coding Techniques: Chapter 1.1. What is Information Theory? Why you should take this course?
Information Theory and Coding Techniques: Chapter 1.1 What is Information Theory? Why you should take this course? 1 What is Information Theory? Information Theory answers two fundamental questions in
More informationCSE468 Information Conflict
CSE468 Information Conflict Lecturer: Dr Carlo Kopp, MIEEE, MAIAA, PEng Lecture 02 Introduction to Information Theory Concepts Reference Sources and Bibliography There is an abundance of websites and publications
More informationDay 15. Tuesday June 12, 2012
Day 15 Tuesday June 12, 2012 1 Properties of Function So far we have talked about different things we can talk about with respect to a function. So far if f : A B we have the following features: A domain:
More informationInstructor (Brad Osgood)
TheFourierTransformAndItsApplications-Lecture17 Instructor (Brad Osgood):Is the screen fading and yes, Happy Halloween to everybody. Only one noble soul here came dressed as a Viking. All right. All right.
More informationThe Transistor. Thomas J. Bergin Computer History Museum American University
The Transistor Thomas J. Bergin Computer History Museum American University In the nineteenth century, scientists were rarely inventors: Samuel F.B. Morse, Alexander Graham Bell, Thomas Alva Edison In
More information1 Ex. 1 Verify that the function H(p 1,..., p n ) = k p k log 2 p k satisfies all 8 axioms on H.
Problem sheet Ex. Verify that the function H(p,..., p n ) = k p k log p k satisfies all 8 axioms on H. Ex. (Not to be handed in). looking at the notes). List as many of the 8 axioms as you can, (without
More informationEntanglement and information
Ph95a lecture notes for 0/29/0 Entanglement and information Lately we ve spent a lot of time examining properties of entangled states such as ab è 2 0 a b è Ý a 0 b è. We have learned that they exhibit
More informationLecture 1: Introduction, Entropy and ML estimation
0-704: Information Processing and Learning Spring 202 Lecture : Introduction, Entropy and ML estimation Lecturer: Aarti Singh Scribes: Min Xu Disclaimer: These notes have not been subjected to the usual
More informationNotes 3: Stochastic channels and noisy coding theorem bound. 1 Model of information communication and noisy channel
Introduction to Coding Theory CMU: Spring 2010 Notes 3: Stochastic channels and noisy coding theorem bound January 2010 Lecturer: Venkatesan Guruswami Scribe: Venkatesan Guruswami We now turn to the basic
More informationGood morning everyone, and welcome again to MSLs World Metrology Day celebrations.
Thank you Gavin.. Good morning everyone, and welcome again to MSLs World Metrology Day celebrations. The aim of this talk is to explain the changes that will be made with the change in the definition of
More informationIntroduction to Information Theory. Part 4
Introduction to Information Theory Part 4 A General Communication System CHANNEL Information Source Transmitter Channel Receiver Destination 10/2/2012 2 Information Channel Input X Channel Output Y 10/2/2012
More informationError Correcting Codes Prof. Dr. P. Vijay Kumar Department of Electrical Communication Engineering Indian Institute of Science, Bangalore
(Refer Slide Time: 00:15) Error Correcting Codes Prof. Dr. P. Vijay Kumar Department of Electrical Communication Engineering Indian Institute of Science, Bangalore Lecture No. # 03 Mathematical Preliminaries:
More informationOutline of the Lecture. Background and Motivation. Basics of Information Theory: 1. Introduction. Markku Juntti. Course Overview
: Markku Juntti Overview The basic ideas and concepts of information theory are introduced. Some historical notes are made and the overview of the course is given. Source The material is mainly based on
More informationText Compression. Jayadev Misra The University of Texas at Austin December 5, A Very Incomplete Introduction to Information Theory 2
Text Compression Jayadev Misra The University of Texas at Austin December 5, 2003 Contents 1 Introduction 1 2 A Very Incomplete Introduction to Information Theory 2 3 Huffman Coding 5 3.1 Uniquely Decodable
More informationMITOCW ocw f99-lec30_300k
MITOCW ocw-18.06-f99-lec30_300k OK, this is the lecture on linear transformations. Actually, linear algebra courses used to begin with this lecture, so you could say I'm beginning this course again by
More informationKnots, Coloring and Applications
Knots, Coloring and Applications Ben Webster University of Virginia March 10, 2015 Ben Webster (UVA) Knots, Coloring and Applications March 10, 2015 1 / 14 This talk is online at http://people.virginia.edu/~btw4e/knots.pdf
More informationLecture 2: Perfect Secrecy and its Limitations
CS 4501-6501 Topics in Cryptography 26 Jan 2018 Lecture 2: Perfect Secrecy and its Limitations Lecturer: Mohammad Mahmoody Scribe: Mohammad Mahmoody 1 Introduction Last time, we informally defined encryption
More informationAlgebraic Codes for Error Control
little -at- mathcs -dot- holycross -dot- edu Department of Mathematics and Computer Science College of the Holy Cross SACNAS National Conference An Abstract Look at Algebra October 16, 2009 Outline Coding
More informationAstronomy 102 Math Review
Astronomy 102 Math Review 2003-August-06 Prof. Robert Knop r.knop@vanderbilt.edu) For Astronomy 102, you will not need to do any math beyond the high-school alegbra that is part of the admissions requirements
More informationMassachusetts Institute of Technology
Name (%): Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Department of Mechanical Engineering 6.050J/2.0J Information and Entropy Spring 2005 Issued: May
More informationImplicit Differentiation Applying Implicit Differentiation Applying Implicit Differentiation Page [1 of 5]
Page [1 of 5] The final frontier. This is it. This is our last chance to work together on doing some of these implicit differentiation questions. So, really this is the opportunity to really try these
More informationExperiment 9. Emission Spectra. measure the emission spectrum of a source of light using the digital spectrometer.
Experiment 9 Emission Spectra 9.1 Objectives By the end of this experiment, you will be able to: measure the emission spectrum of a source of light using the digital spectrometer. find the wavelength of
More informationNoisy channel communication
Information Theory http://www.inf.ed.ac.uk/teaching/courses/it/ Week 6 Communication channels and Information Some notes on the noisy channel setup: Iain Murray, 2012 School of Informatics, University
More informationSimple Interactions CS 105 Lecture 2 Jan 26. Matthew Stone
Simple Interactions CS 105 Lecture 2 Jan 26 Matthew Stone Rutgers University Department of Computer Science and Center for Cognitive Science Themes from the Preface 2 Computers are multimedia devices Text
More informationUsing Microsoft Excel
Using Microsoft Excel Objective: Students will gain familiarity with using Excel to record data, display data properly, use built-in formulae to do calculations, and plot and fit data with linear functions.
More informationSample. Contents SECTION 1: PLACE NAMES 6 SECTION 2: CONNECTING TO PLACES 21 SECTION 3: SPACES: NEAR AND FAR 53
Contents Teachers' Notes 4 National Curriculum Links 5 SECTION 1: PLACE NAMES 6 Teachers' Notes 7-8 Activities Names Of Places 9 Place Names Are Important 1 10 Place Names Are Important 2 11 The Meanings
More informationUnderstanding The Law of Attraction
Psychic Insight 4-5-17 New Blog Below Understanding The Law of Attraction Everything in the universe is energy and vibrates at a distinct frequency. Understanding that these energies are attracted to similar
More informationIntroduction to Information Theory. Uncertainty. Entropy. Surprisal. Joint entropy. Conditional entropy. Mutual information.
L65 Dept. of Linguistics, Indiana University Fall 205 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission rate
More informationInformation Theory, Statistics, and Decision Trees
Information Theory, Statistics, and Decision Trees Léon Bottou COS 424 4/6/2010 Summary 1. Basic information theory. 2. Decision trees. 3. Information theory and statistics. Léon Bottou 2/31 COS 424 4/6/2010
More informationAsk. Don t Tell. Annotated Examples
Ask. Don t Tell. Annotated Examples Alfonso Gracia-Saz (alfonso@math.toronto.edu) The three principles: 1. Where is the student? 2. Help minimally. 3. Ask. Don t Tell. Ask. Don t Tell. 1 BRETT 1 Brett
More information9. Distance measures. 9.1 Classical information measures. Head Tail. How similar/close are two probability distributions? Trace distance.
9. Distance measures 9.1 Classical information measures How similar/close are two probability distributions? Trace distance Fidelity Example: Flipping two coins, one fair one biased Head Tail Trace distance
More informationDept. of Linguistics, Indiana University Fall 2015
L645 Dept. of Linguistics, Indiana University Fall 2015 1 / 28 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission
More informationTo Infinity and Beyond
To Infinity and Beyond 22 January 2014 To Infinity and Beyond 22 January 2014 1/34 In case you weren t here on Friday, Course website: http://sierra.nmsu.edu/morandi/math210 Get a copy of the syllabus
More informationDEPARTMENT OF EECS MASSACHUSETTS INSTITUTE OF TECHNOLOGY. 6.02: Digital Communication Systems, Fall Quiz I. October 11, 2012
6.02 Fall 2012, Quiz 2 Page 1 of 12 Name: DEPARTMENT OF EECS MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.02: Digital Communication Systems, Fall 2012 Quiz I October 11, 2012 your section Section Time Recitation
More informationMeasurement Error PHYS Introduction
PHYS 1301 Measurement Error Introduction We have confidence that a particular physics theory is telling us something interesting about the physical universe because we are able to test quantitatively its
More informationBasic methods to solve equations
Roberto s Notes on Prerequisites for Calculus Chapter 1: Algebra Section 1 Basic methods to solve equations What you need to know already: How to factor an algebraic epression. What you can learn here:
More informationDirect Proof and Counterexample I:Introduction
Direct Proof and Counterexample I:Introduction Copyright Cengage Learning. All rights reserved. Goal Importance of proof Building up logic thinking and reasoning reading/using definition interpreting :
More informationTake the measurement of a person's height as an example. Assuming that her height has been determined to be 5' 8", how accurate is our result?
Error Analysis Introduction The knowledge we have of the physical world is obtained by doing experiments and making measurements. It is important to understand how to express such data and how to analyze
More informationMITOCW ocw f99-lec01_300k
MITOCW ocw-18.06-f99-lec01_300k Hi. This is the first lecture in MIT's course 18.06, linear algebra, and I'm Gilbert Strang. The text for the course is this book, Introduction to Linear Algebra. And the
More informationMajor upsetting discoveries: Today s Objectives/Agenda. Notice: New Unit: with Ms. V. after school Before Friday 9/22.
Bellwork Monday 9 18 17 Tuesday 9 19 17 *This should be the last box on bellwork: 1. What are some major points in history where common knowledge was upset by a new discovery? 2. Draw what you think an
More informationIntroduction to Informatics
Introduction to Informatics Lecture 7: Modeling the World Readings until now Lecture notes Posted online @ http://informatics.indiana.edu/rocha/i101 The Nature of Information Technology Modeling the World
More informationDirect Proof and Counterexample I:Introduction. Copyright Cengage Learning. All rights reserved.
Direct Proof and Counterexample I:Introduction Copyright Cengage Learning. All rights reserved. Goal Importance of proof Building up logic thinking and reasoning reading/using definition interpreting statement:
More informationX 1 : X Table 1: Y = X X 2
ECE 534: Elements of Information Theory, Fall 200 Homework 3 Solutions (ALL DUE to Kenneth S. Palacio Baus) December, 200. Problem 5.20. Multiple access (a) Find the capacity region for the multiple-access
More informationTo Infinity and Beyond. To Infinity and Beyond 1/43
To Infinity and Beyond To Infinity and Beyond 1/43 Infinity The concept of infinity has both fascinated and frustrated people for millennia. We will discuss some historical problems about infinity, some
More informationPig organ transplants within 5 years
www.breaking News English.com Ready-to-use ESL / EFL Lessons Pig organ transplants within 5 years URL: http://www.breakingnewsenglish.com/0509/050911-xenotransplant.html Today s contents The Article 2
More informationBoolean Algebra & Digital Logic
Boolean Algebra & Digital Logic Boolean algebra was developed by the Englishman George Boole, who published the basic principles in the 1854 treatise An Investigation of the Laws of Thought on Which to
More information