deconvolution Documentation

Transcription

1 deconvolution Documentation Release Frederic Grabowski, Paweł Czyż Aug 07, 2018

2

3 Contents 1 Readme deconvolution Indices and tables 11 Bibliography 13 i

4 ii

5 CHAPTER 1 Readme 1.1 deconvolution A Python module providing Deconvolution class that implements and generalises Ruifrok-Johnston color deconvolution algorithm [RJ], [IJ]. It allows one to split an image into distinct color layers in just a few lines of code: from deconvolution import Deconvolution from PIL import Image img = Image.open("image.jpg") # Declare an instance of Deconvolution, with image loaded and with color basis defining what layers are interesting decimg = Deconvolution(image=img, basis=[[1, 0.1, 0.2], [0, 0.1, 0.8]]) # Constructs new PIL Images, with different color layers layer1, layer2 = decimg.out_images(mode=[1, 2]) Installation You can install the package using pip: pip install deconvolution Alternatively, you can clone the repository and run: make install Since then you can import use the module in your scripts: from deconvolution import Deconvolution d = Deconvolution() 1

6 1.1.2 Testing # For Python 3 users make test # For Python 2 users make comp # Check the code coverage make coverage # Check the coverage interactively, using a web browser make html Deconvolve For better usage experience we created a script allowing one to deconvolve images from the shell. Copy deconvolve.py file into /usr/local/bin or, if you want to use it locally: mkdir ~/bin cp deconvolve.py ~/bin export PATH=~/bin:$PATH Since then you can deconvolve images using: deconvolve.py image1.png image2.png... # For help deconvolve.py -h Documentation Check out our documentation at Read The Docs Contributors Method developed by Frederic Grabowski generalising Ruifrok-Johnston algorithm [RJ]. and implemented by Frederic Grabowski [FG] and Paweł Czyż [PC]. We are very grateful to prof. Daniel Wójcik and dr Piotr Majka [N1], [N2] who supervised the project. We also would like to thank prof. Gabriel Landini [GL], who implemented the colour deconvolution in ImageJ [IJ] and allowed us to test the algorithm on his data References Examples Ruifrok-Johnston deconvolution Assume that we have an image and we know three stains. We can deconvolve image and get density layers: """ In this example, we have an image and a known basis with three vectors. We want to get the density layers. 2 Chapter 1. Readme

7 In this example we use a cropped image [1], released under CC licence. [1] File:Chronic_lymphocytic_leukemia_-_high_mag.jpg """ from deconvolution import Deconvolution from PIL import Image def join_horizontally(*args): """Joins many PIL images of the same dimensions horizontally""" w, h = args[0].size n = len(args) joined = Image.new("RGB", (n*w, h)) for x_off, img in zip(range(0, n*w, w), args): joined.paste(img, (x_off, 0)) return joined if name == " main ": # Load an image original = Image.open("cropped.jpg") # Create a deconvolution object with the image dec = Deconvolution(image=original, basis=[[0.91, 0.38, 0.71], [0.39, 0. 47, 0.85], [1, 0, 0]]) # Produce density matrices for both colors. Be aware, as Beer's law do not always hold. first_density, second_density, third_density = dec.out_scalars() print(first_density.shape, second_density.shape, third_density.shape) # Produce reconstructed image, first layer, second layer, third layer and rest out_images = dec.out_images(mode=[0, 1, 2, 3, -1]) # Original image, reconstructed, layers and rest join_horizontally(original, *out_images).show() Two stain deconvolution Alternatively, we can use two stains. If we know them, we can just mimic the procedure for two stains: """ In this example, we have an image and a known basis with two vectors. We want to get the density layers. In this example we use a cropped image [1], released under CC licence. [1] File:Chronic_lymphocytic_leukemia_-_high_mag.jpg """ from deconvolution import Deconvolution from PIL import Image 1.1. deconvolution 3

8 def join_horizontally(*args): """Joins many PIL images of the same dimensions horizontally""" w, h = args[0].size n = len(args) joined = Image.new("RGB", (n*w, h)) for x_off, img in zip(range(0, n*w, w), args): joined.paste(img, (x_off, 0)) return joined if name == " main ": # Load an image original = Image.open("cropped.jpg") # Create a deconvolution object with the image dec = Deconvolution(image=original, basis=[[0.91, 0.38, 0.71], [0.39, 0. 47, 0.85]]) # Produce density matrices for both colors. Be aware, as Beer's law do not always hold. first_density, second_density = dec.out_scalars() print(first_density.shape, second_density.shape) # Produce reconstructed image, first layer, second layer and rest out_images = dec.out_images(mode=[0, 1, 2, -1]) # Original image, reconstructed, layers and rest join_horizontally(original, *out_images).show() But if we do not know at least one of them, we should first find appropriate vectors: """ In this example, we have an image and an insufficient number of color vectors - we will use Deconvolution complete_basis method to find other vectors. Then we try to make them more independent applying resolve_dependencies with different belligerency parameter. We show all the bases found. In this example we use a cropped image [1], released under CC licence. [1] File:Chronic_lymphocytic_leukemia_-_high_mag.jpg """ from deconvolution import Deconvolution from PIL import Image def join_horizontally(*args): """Joins many PIL images of the same dimensions horizontally""" w, h = args[0].size n = len(args) joined = Image.new("RGB", (n*w, h)) for x_off, img in zip(range(0, n*w, w), args): 4 Chapter 1. Readme

9 joined.paste(img, (x_off, 0)) return joined def join_vertically(*args): """Joins many PIL images of the same dimensions vertically""" w, h = args[0].size n = len(args) joined = Image.new("RGB", (w, n*h)) for y_off, img in zip(range(0, n*h, h), args): joined.paste(img, (0, y_off)) return joined if name == " main ": # Load an image original = Image.open("cropped.jpg") # Create a deconvolution object with the image dec = Deconvolution(image=original, sample_density=6) # Complete basis - as we did not provide 2 or 3 vectors, it needs to be found dec.complete_basis() # We can get the basis found print("basis before resolve:\n{}\n".format(dec.pixel_operations.get_ basis())) # Produce reconstructed image, first layer, second layer and rest out_images1 = dec.out_images(mode=[0, 1, 2, -1]) # Original image, reconstructed, layers and rest before_resolve = join_horizontally(original, *out_images1) # Resolve dependencies - make the vectors more independent dec.resolve_dependencies(belligerency=0.1) # We can get the basis found print("basis after resolve:\n{}\n".format(dec.pixel_operations.get_ basis())) # And produce reconstructed image, first layer, second layer and rest out_images2 = dec.out_images(mode=[0, 1, 2, -1]) # Construct a joined image from original image, reconstructed, layers and rest after_resolve = join_horizontally(original, *out_images2) # Resolve dependencies with huge belligerency - make the vectors very independent dec.resolve_dependencies(belligerency=1) print("basis after aggressive resolve:\n{}\n".format(dec.pixel_operations. get_basis())) # Produce reconstructed image, first layer, second layer and rest out_images3 = dec.out_images(mode=[0, 1, 2, -1]) # Show original image, reconstructed, layers and rest after_huge_resolve = join_horizontally(original, *out_images3) # Show all images for visual comparision join_vertically(before_resolve, after_resolve, after_huge_resolve).show() 1.1. deconvolution 5

10 Mathematical description of color deconvolution Introduction Let us imagine a light source that sends light through layers of different substances - each of them absorbing some of the light passing through. Given a digital recording of the light that passed through this setup (and information of the input light), we would like to reconstruct the layout of the substances. Each pixel is represented by a three-component vector (it s RGB components). Hadamard (or entry-wise) product: x 1 y 1 x 1 y 1 x 2 y 2 = x 2 y 2 x 3 y 3 x 3 y 3 I will make frequent use of the The Hadamard product is similar to the standard multiplication of real numbers and I will use the same notation for inverses and powers. Consider light represented by a RGB-vector i passing through a unit-wide layer of some substance. I will assume that Beer s law [LB] hold s, that is the RGB-vector of the outgoing light will be: i s = i 1s 1 i 2 s 2, i 3 s 3 where s is specific for this substance (the choice of unitary width does change this value; this issue will be addressed later on). If, for example, s = (1, 1, 1) the layer does not absorb any light, for s = (0, 1, 1) the red channel is completely absorbed, but the remaining channels are untouched. Should the layer be a times wider, the RGB components of the outgoing light would be: i 1 s a i s a 1 = i 2 s a 2, i 3 s a 3 for multiple substances, i 1 p a i p a q b 1 q1 b = i 2 p a 2 q2 b.... i 3 p a 3 q3 b... This equation implies that changing the order of layers or splitting some of them does not change the outgoing light. I will work under the assumption that this also holds for mixed substances Ruifrok-Johnston deconvolution In the case studied by Ruifrok and Johnston [RJ], the light i passes through three substances. Vectors (,,) describing absorption rates for all substances (that is, absorption coefficients for unit-wide substance layers) are assumed to be known. The width of each substance layer (which may change from point to point) has to be calculated given the output light. Suppose the camera registers a single vector r at some arbitrary pixel. We wish to express this vector according to the equation: i v a u b w c = r. Solving this equation for a, b and c gives the layers widths in unit lengths. We may then compute how much light would have passed through each layer separately: the first deconvolution is just i v a, the second i u b and the third i w c. It is possible that no real non-negative a, b, c solve this equation (due to data noise, imperfect digitization, 6 Chapter 1. Readme

11 traces of other substances, etc.). In that case, the reconstructed image will differ from the original: this difference can be visualized by considering rest picture : r v a u b w c. Let us briefly return to the issue of picking a unit width. Notice that changing the unit width of a substance by a factor of λ, changes the constant $a$ at each pixel to a/λ, and thus the density distribution a times the unit width does not change. Similarly, the deconvolved images don t change: i v a = i ( v λ ) a/λ. Equation (ref{eq:decon3}) is in fact a set of three equations - one for each component. Lets rewrite it for the k-th component and transform equivalently: v a k u b k w c k = r k i k a log v k + b log u k + c log w k = log(r k /i k ), going back to vector representation: log v 1 log u 1 log w 1 log v 2 log u 2 log w 2 a b log(r 1/i 1 ) log(r 2 /i 2 ) = 0. log v 3 log u 3 log w 3 c log(r 3 /i 3 ) This equation is solvable if and only if the left matrix is invertible, and i k 0. In physical terms, the first assumption states that no substance can be faked by mixing the remaining two, and the second that the input light is nonzero for all channels. Given an input image, at each pixel r we may solve for a, b, c. Should we get any negative results, it means that this particular pixels color can not be obtained by mixing the given substances. In this case the assumption that the image was obtained by mixing the given substances is violated - hence it s reasonable to disregard such results data noise and drop all negative parts. Having that done, we are able to construct five pixels: Reconstructed pixel: i v a u b w c Difference from the original image, due to negative cut-off: r v a u b w c Three single-substance pixels: + i v a + i u b + i w c After processing every pixel in this manner, the reconstructed image, three single substance images, and one remainder image (showing the error) are obtained. An example is shown in Figure ref{fig:ruifrok}. Until now the approach was based on Ruifrok and Johnston - however, the choice of formalism makes it easier to look for further development. Handle any number of channels straightforward: in fact, for the general case, the notation does not change. Secondly, if only two substances are of interest, Ruifrok and Johnston suggest measuring the absorbances of those two substances, and then choosing the third so that it minimizes the negative cut-off. The third single-substance image is then used as a measure of error. This arbitrarity seems a bit artificial - now I introduce a method developed by Frederic Grabowski Two-substance deconvolution Choosing the third substance by hand so that it minimizes data loss due to negative cut-offs introduces ambiguity into the measurement, and seems artificial. In order to fix this problem, drop the third substance entirely, and look for a, b that minimize the squared error of the approximation: [ log(r a 1 /i 1 ) b] log v 1 log u 1 log v 2 log u 2 log v 3 log u 3 log(r 2 /i 2 ) log(r 3 /i 3 ) deconvolution 7

12 Where both the matrix and the 3-vector are given. Clean up the notation: inf f(x) = inf A x y 2 x R 2 x R2 We want to know the x for which this infimum is obtained. This x always exists, because it is the orthogonal projection of y onto A(R 2 ). For each minimum grad f = 0. That is, for the first component: 0 = 1 f (x 1, x 2 ) = 2 x 1 x 1 3 A k1 (A k1 x 1 + A k2 x 2 r k ) k=1 3 A 2 k1 + x 2 k=1 3 A k1 A k2 = k=1 (1.1) 3 A k1 r k Combining (ref{eq:solab}) with the equation for the second component we get a set of two linear equations: [ ] [ ] [ A 2 k1 A k1 A k2 x1 = k A ] k1r k A k1 A k2 x 2 k A. k2r k k The equation above can be easily solved if and only if it s determinant is not 0: [ det k A2 k1 k A ] k1a k2 k A = k1a k2 k A2 k2 A 2 k1 ( ) 2 A 2 k2 A k1 A k2 0 k k A 2 k2 The Cauchy-Schwarz inequality states that the considered determinant is 0 if and only if there is a number t for which A k1 = t A k2. This is again the mixing independence of the basis. If dim A(R 2 ) < 2, then there does not exist a unique x for which the projected vector is obtained. We now have a method for finding the best a, b solving equation (ref{eq:2vec}). This means that for each pixel and a basis of two given substances, we are able to calculate four pixels: - Best reconstructed pixel: i v a u b - Difference from the original image: r v a u b - Two single-substance pixels: i v a and i u b After processing every pixel in this manner the reconstructed image, two single substance images, and one remainder image (showing the error) are obtained. An example is shown in Figure ref{fig:auto}. k k= Formulation of the optimization problem Considering deconvolutions with two substances has another advantage - it gives a criterium for comparing bases. Taking inf LHS of eq.(1.1.10) 2 a,b,c R p pixels does not work, because the equation is always soluble and the expression is identically zero (at least for all independent bases). Decreasing the number of degrees of freedom (that is, the number of substances to match) solves this difficulty: log v 1 log u 1 [ ] log(r log v 2 log u 2 a 1 /i 1 ) log(r b 2 /i 2 ) 0. log v 3 log u 3 log(r 3 /i 3 ) log v 1 log u 1 [ log(r inf log v 2 log u 2 a 1 /i 1 ) log(r a,b R b] 2 /i 2 ) 2 (1.2) p pixels log v 3 log u 3 log(r 3 /i 3 ) 8 Chapter 1. Readme

13 To solve this optimization problem, first clean up the notation. Let: A = a 11 a 12 a 21 a 22, a 31 a 32 [ ] [ x1 a = x = x 2 b] y 1 (p) log(r 1 /i 1 ) y 2 (p) = y(p) = log(r 2 /i 2 ) y 3 (p) log(r 3 /i 3 ) Given some y(p), the problem is to find a 2 3 matrix A, that minimizes the expression: f(a) = inf A x y(p) 2 x R2 p pixels Solving the optimization problem For any A in equation ref{eq:optorig}: inf A x x R y(p) 2 = d( y(p), A(R 2 )) 2, 2 hence we want to minimize the mean squared distance of the points $y$ from the image space of A. There are two cases: either dim A(R 2 ) = 2 or is strictly less than 2. In the second case, we are always able to choose such a matrix A, that the previous image is a subspace of the new image, but then the distances can only be smaller. It thus suffices to find the best two dimensional space. Every such space has a normal vector, which we choose so that the third component in non-negative (this convention is arbitrary, but does not matter). We can now rewrite (ref{eq:optorig}) as: f(a) = g( n) = n y(p) 2 p pixels Any n minimizing (ref{eq:optg}), determines a class of matrices minimizing (ref{eq:optorig}) - precisely those, whose image is perpendicular to n. We only need to consider n such that n = 1. Because the set of such n is compact in R 3, we can apply the method of Lagrange multipliers: (g( n) λ n 2 ) = 0, after expanding and rearranging p pixels ( y(p) n ) y(p) = λ n. The left hand side is a linear operator from R 3 to R 3 applied to n. Equation (ref{eq:eigenorig}) is just the eigenvalue equation for this operator. Moreover, multiplying both sides by n we notice λ = g( n). The smallest-eigenvalue eigenvector is the n minimizing g( n), and the corresponding eigenvalue the value g( n) Computing the deconvolution Rewrite the left hand side of equation (ref{eq:eigenorig}) using index notation: p pixels 3 y i (p)y j (p) n i, i= deconvolution 9

14 is the j-th component of the resulting vector. Hence the matrix of the linear transformation is: (Y ) ij = p pixels 3 y i (p)y j (p) i=1 Given an input image, first calculate Y, and find it s eigenvalue decomposition. Pick the eigenvector n with smallest eigenvalue. Choosing an A such that n is perpendicular to A s image is equivalent to choosing a basis with both elements perpendicular to n. To have any preference, let s return to the physical interpretation. Naively, all these bases allow us to mix the same set of colors: but not for all bases will this mixing will be physically meaningful. Consider the following example: the basis consisting of two substances, one absorbing only red the other only blue, will be equivalent to a basis of one substance absorbing both red and blue, and the other only blue. However, only in the first base is it possible to construct a color with only the red channel absorbed. This happens because we cannot physically have negative widths of substances. It seems advantageous to choose the basis that allows us to mix the widest range of colors physically. It turns out that this choice is not always optimal. For now, we stick with the maximal physical color range. The basis of our choosing is the one for which both vectors are non-negative (so that the resulting substances absorb light, and not amplify it), and have the biggest angle between them. This determines them uniquely up to a rearrangement References 10 Chapter 1. Readme

15 CHAPTER 2 Indices and tables genindex modindex search 11

16 12 Chapter 2. Indices and tables

17 Bibliography [RJ] Research paper by Ruifrok and Johnston [IJ] ImageJ webpage [N1] Laboratory of Neuroinformatics webpage [GL] Prof. Gabriel Landini s webpage [N2] [FG] [PC] [LB] Modern analysis of this law can be found in Employing Theories Far beyond Their Limits The Case of the (Boguer-) Beer Lambert Law by Mayerhoefer et al. [RJ] Research paper by Ruifrok and Johnston 13