Exam. Matrikelnummer: Points. Question Bonus. Total. Grade. Information Theory and Signal Reconstruction Summer term 2013

Transcription

1 Exam Name: Matrikelnummer: Question Bonus Points Total Grade 1/6

2 Question 1 You are traveling to the beautiful country of Markovia. Your travel guide tells you that the weather w i in Markovia on a particular day i is sunny, w i = s, for 80% of all days or it is cloudy, w i = c, for 20% of all days. There are no other weather conditions in Markovia and the weather changes only during nights. The probability for a weather change is 10% if it is sunny, P (w i+1 = c w i = s) = 0.1, (1) and 40% if it is cloudy, P (w i+1 = s w i = c) = 0.4, (2) irrespective of what it has been on earlier days, P (w i+1 w i, w i 1, w i 2,...) = P (w i+1 w i ). a) You arrive on a sunny day, w i = s, in Markovia. Calculate the probability that it was cloudy /6 there the day before, P (w i 1 = c w i = s). b) What is the total probability for a weather change P (w i+1 w i w i ) in Markovia during an /5 arbitrary night? c) The Markovian weather forecast for some day i predicts a sunshine probability of /5 p i = P (w i = s forecast). (3) What is the sunshine probability there for the following day, p i+1 = P (w i+1 = s forecast)? (4) d) Verify or correct the travel guide s statement on the frequency of 80% sunny and 20% cloudy /4 days in Markovia. Hint: Your result of question c) might be useful for this. 2/6

3 Question 2 Consider the following coin toss experiment: A number n of coin tosses are performed and the results are stored in a data vector d (n) = (d 1,..., d n ) {0, 1} n, where d i = 1 represents head and d i = 0 tail. Individual tosses are independent from each other. All tosses are done with the same coin with an unknown bias λ [0, 1], i.e., P(d i λ) = λ di (1 λ) 1 di (5) We do, however, have the prior knowledge that extreme biases are unlikely, formalized by the prior for the bias parameter P(λ) = 6λ (1 λ). (6) Assume that k out of the n coin tosses yielded head, i.e., k = n i=1 d i. Since all tosses are independent, the order of heads and tails does not contain any information on the bias parameter λ. So instead of considering the whole data vector d (n), we can use the sufficient statistic k. Hint: 1 0 dx xα (1 x) β = α!β! (α+β+1)! α, β N \ {0} ( ) a You may want to use = b a! b!(a b)! for a, b N and a b. a) Calculate the likelihood P (k λ), i.e., the probability of obtaining k heads in n tosses, given a /4 bias λ. b) Derive the exact posterior probability distribution P(λ k). /6 c) Use this result to calculate the posterior mean λ = λ P(λ k) of the bias parameter λ. /5 d) Now calculate the posterior variance σ 2 λ = (λ λ) 2 P(λ k) to see that not all biases can be estimated with the same accuracy. as a function of λ and n in order /5 3/6

4 Question 3 Consider a real-valued signal field s with a Gaussian prior, P(s) = G(s, S), (7) that is observed with an instrument that exhibits an almost linear response, d = R ( s + rs 2) + n. (8) Here, R is a linear operator, r R with r 1 is a small parameter that determines the strength of the nonlinearity in the instrumental response, s 2 denotes the local squaring of the signal field, i.e., ( s 2) x = (s x) 2, and n is additive Gaussian noise, i.e., P(n) = G(n, N). (9) a) Consider first the case of an exactly linear response, i.e., r = 0. Derive the Hamiltonian /4 H(d, s) = log (P(d, s)) (10) for this problem. You may drop all terms that do not depend on s. b) Show that the posterior probability density in the case with r = 0 is of Gaussian form, i.e., /4 P(s d) = G(s m 0, D), and derive expressions for its mean and covariance, m 0 = s P(s d) and D = (s m 0 )(s m 0 ) P(s d), (11) as a function of d, S, N, and R. c) Now consider the case with small but non-zero r. Calculate the Hamiltonian in this case and /6 write it in the form H(s, d) = H 0 j s s D 1 s + k=2 1 k! Λ(k) x 1x 2 x k s x1 s x2 s xk, (12) where only the coefficients Λ (k) depend on r and we use the convention that repeated indices are integrated over. Give expressions for j, D, and all non-zero Λ (k). You do not need to calculate H 0. d) Write down the diagrammatic expansion of the partition function log (Z(d)) up to linear order /3 in r. e) Find the diagrammatic expressions for the posterior mean and covariance, /3 m r = s P(s d) up to first order in r. and (s mr )(s m r ) P(s d), (13) 4/6

5 Question 4 Consider the potential V ( r) which is symmetric with respect to the radial distance r = r, V (r) = a r + b. (14) This potential is parametrized by the unknown numbers a, b R and can be measured at strictly positive radii, i.e., r > 0. Furthermore, only a single data point d R can be obtained, d = V (r) + n, (15) where the noise n is assumed to obey a Gaussian statistic P(n) = G(n, N). The noise variance N = N(r), however, depends on the measurement position, N(r) = r (16) Hint: Remember the calculus for 2 2 matrices, ( ) A B = AD BC and C D ( ) 1 A B 1 = C D AD BC ( D B C A a) An information entropy S B given as /5 S B = Ds P(s d) log P(s d), (17) describes the lack of knowledge in the complete signal phase space of the posterior. Derive a general expression for the information entropy S B for a Gaussian posterior P(s d) = G(s m, D) with mean m and covariance D. ( ) a b) Consider the signal s =, for which a Gaussian prior P(s) = G(s, 1) can be assumed. /4 b Write Eq. (15) in the form d = R s + n and give R explicitly. Work out an expression for the joint probability P(d, s) and calculate the corresponding Hamiltonian H(d, s) = log P(d, s) = 1 2 s D 1 s j s + H 0. You may drop H 0. Identify the information source j and the inverse information propagator D 1. c) You verified in a) that information entropy S B = S B (D) is a monotonically increasing function /3 of D. Find the best position r to estimate both, a and b, by minimizing D from b) with respect to r. d) Now, consider the signal s = a for which b becomes a nuisance parameter. /4 Work out an expression for the joint probability P(d, a), and calculate the corresponding Hamiltonian H(d, a) = log P(d, a) = 1 2 D 1 a 2 ja + H 0. You may drop H 0. Identify the information source j and the information propagator D. e) Find the best position r a to estimate a irrespectively of b, by minimizing D from d) with respect /3 to r. f) Guess at which radius r b one should measure in order to obtain the most certain estimate for /1 the parameter b. No justification required. ). 5/6

6 Question 5 A temporally varying radio signal s = (s t R) t is detected with a radio receiver. The receiver amplifies the signal via a resonant circuit, for which the voltage u = (u t R) t follows the equation of a signal-driven, but weakly damped oscillator: d 2 u t dt 2 + 2η du t dt + u t = s t. (18) Here, problem specific and therefore dimensionless time units were chosen, so that the small damping constant η (0, 1/ 2) is dimensionless. The receiver electronics adds some statistically homogeneous white Gaussian noise n = (n t R) t with known variance n t n t P(n) = σ 2 δ(t t ) to the output data d t = u t + n t. (19) Please use the Fourier sign conventions s ω = dt e iωt s t and s t = dω 2π e+iωt s ω or note explicitely the ones you use. Factors of 2π may be neglected in your calculation. a) Work out the signal response of the voltage in the Fourier domain, i.e., give R ωω of /4 dω u ω = 2π R ωω s ω. (20) b) Work out the signal response of the voltage in the time domain, i.e., give R tt of /7 u t = dt R tt s t. (21) c) Suppose the signal is a statistically homogeneous Gaussian random field with known power /3 spectrum P s (ω) = s ω 2. In order to assess how well the signal can be reconstructed at P(s) each frequency from the output data, calculate the uncertainty spectrum sω P D (ω) = s ω P(s d) 2. (22) P(s d) d) Now, suppose you want to design a matching transmitter to send an amplitude (modulated) /4 signal to the receiver. The transmitter just produces (approximately) a fixed amount of signal power p = P s (ω ) at a single, tunable frequency ω. What is the optimal transmission frequency ω to permit the most accurate signal reconstruction on the receiver side? Assume that p as well as ω are known to the receiver. e) Calculate the complex eigenfrequency ω 0 of the undriven oscillator. Set s = 0 in Eq. (18) for /2 this. Is the eigenfrequency the same as the optimal transmission frequency ω? 6/6