Nanostructure Science and Technology Series Editor: David J Lockwood, FRSC National Research Council of Canada Ottawa, Ontario, Canada For further volumes: http://www.springer.com/series/6331 Thomas Vogt • Wolfgang Dahmen • Peter Binev Editors Modeling Nanoscale Imaging in Electron Microscopy 123 Editors Thomas Vogt NanoCenter and Department of Chemistry and Biochemistry University of South Carolina 1212 Greene Street Columbia, SC 29208 USA Wolfgang Dahmen Institut fă r Geometrie u und Praktische Mathematik Department of Mathematics RWTH Aachen 52056 Aachen Germany Peter Binev Department of Mathematics and Interdisciplinary Mathematics Institute University of South Carolina 1523 Greene Street Columbia, SC 29208 USA ISSN 1571-5744 ISBN 978-1-4614-2190-0 e-ISBN 978-1-4614-2191-7 DOI 10.1007/978-1-4614-2191-7 Springer New York Dordrecht Heidelberg London Library of Congress Control Number: 2012931557 © Springer Science+Business Media, LLC 2012 All rights reserved This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com) Preface Imaging with electrons, in particular using scanning transmission electron microscopy (STEM), will become increasingly important in the near future, especially in the materials and life sciences Understanding cellular interaction networks will enable transformative research such as “visual proteomics,” where spatial arrangements of the proteome or particular subsets of proteins will be mapped out In the area of heterogeneous catalysis, which in many cases relies on nanoparticles deposited onto supports recently, achieved advances in imaging and characterization of catalysts and precatalysts are transforming the field and allowing more and more rational design of multifunctional catalysts Advances in nanoscale manufacturing will require picometer resolution and control as well as the elimination of routine visual inspection by humans to become viable and implemented in “real” manufacturing environments There are (at least) two major obstructions to fully exploit the information provided by electron microscopy On the one hand, a major bottleneck in all these applications is currently the “human-in-the-loop” resulting in slow and labor-intensive selection and accumulation of images A “smart” microscope in which instrument control, image prescreening, image recognition, and machine learning techniques are integrated would transform the use of electron imaging in materials science, biology, and other fields of research by combining fast and reliable imaging with automated highthroughput analysis such as combinatorial chemical synthesis in catalysis or the multiple “omics” in biology On the other hand, even if environmental perturbations could be completely avoided a principal dilemma remains that results from the fact that the acquired images offer only an “ambiguous reflection” of reality due to inherently noisy data and this is the primary issue addressed in this volume The noise structure is highly complex and far from fully being understood In particular, it depends in a complex way on the electron dose deployed per unit area Low noise levels require a high dose that, in turn, may cause damage In most cases, highenergy electrons damage biological and organic matter and thus require special techniques for imaging when using electron microscopes with beams in the 100–300 kV range Experiments are frequently performed at “nonbiological” temperatures v vi Preface (i.e., cryo-electron microscopy) to reduce damage But even when investigating inorganic material at the atomic resolution level, relatively low dose image acquisition is often required to avoid damaging the sample This again impacts significantly the signal-to-noise ratio of the resulting images The required low doses necessitate new paradigms for imaging, more sophisticated data “denoising” and image analysis as well as simulation techniques In combination with ongoing experimental work to reduce the environmental impact during nano-imaging experiments (e.g., vibrations, temperature, acoustic, and electromagnetic interference), we have begun to develop and apply nonlinear probabilistic techniques They are enhanced by learning theory to significantly reduce noise by systematically exploiting repetitive similarities of patterns within each frame as well as across a series of frames combined with new registration techniques Equating “low electron dose” with “few measurements” is an intriguing idea that is going to radically alter image analysis—and even acquisition—using techniques derived from “Compressed Sensing,” an emerging new paradigm in signal processing A key component here is to use randomness to extract the essential information from signals with “sparse information content” by reducing the number of measurements in ranges where the signal is sparse Working first with inorganic materials allows us to validate our methods by selecting on the basis of high-resolution images an object to be imaged at lower resolution Building on the insight gained through these we can then proceed to image silicate or organic materials which cannot be exposed to high energy electrons for extended periods of time Examples of such an approach are given in Chap Part of our work has greatly benefitted from three workshops organized at the University of South Carolina by the Interdisciplinary Mathematics Institute and the NanoCenter entitled “Imaging in Electron Microscopy” in 2009 and 2010 and “New Frontiers in Imaging and Sensing” in 2011 At these workshops world-class practitioners of electron microscopy, engineers, and mathematicians began to discus and initiate innovative strategies for image analysis in electron microscopy The goal of our work is to develop and apply novel methods from signal and image processing, harmonic analysis, approximation theory, numerical analysis, and learning theory Simulation is an important and necessary component of electron image analysis in order to assess errors of extracted structural parameters and better understand the specimen–electron interactions It thereby helps improve the image as well as calibrate and assess the electron optics and their deviations due to environmental effects such as acoustic noise, temperature drifts, radio-frequency interferences, and stray AC and DC magnetic fields The intuition-based approach based on Z2 -contrast can be misleading if for instance in certain less compact structures electron channeling effects are not correctly taken into account Over the last years, we have established a global research collaboration anchored around electron microscopists at USC (Thomas Vogt, Douglas Blom) and other people such as Angus Kirkland (Oxford), Nigel Browning (UC Davis and LLNL) with mathematicians at USC’s Interdisciplinary Mathematics Institute (Peter Binev, Robert Sharpley), Ronald DeVore (Texas A&M) and Wolfgang Dahmen (RWTH Aachen) These collaborations are critical in exploring novel denoising, Preface vii nonlocal algorithms as well as new methods to exploit Compressed Sensing for nanoscale chemical imaging This book is to be seen as a progress report on these efforts We thought it was helpful to have Professor Michael Dickson (Philosophy, University of South Carolina) address issues of realism and perception of nanoimages and how we might think of them in a “Kantian” way Chapters and are from well-established practitioners in the field of scanning transmission electron microscopy, led by Professors Nigel Browning and Angus Kirkland from the University of California Davis and Oxford University, respectively Both chapters exemplify what it means to “image at the edge” and push the method to its current limitations Limitations that might be pushed back a bit further using different image analysis techniques Chapters and rely heavily on two facilities at USC: many experimental data were taken on a JEOL JEM-2100F (200 kV) microscope with field emission gun, spherical aberration corrector, STEM mode, High Angle Annular Dark Field detector (HAADF), EELS, EDX, and tomography mode This instrument provides routinely sub-Angstrom image resolution and elemental resolution at the atomic level and is operated by Dr Douglas Blom Second, we have a state-of-theart floating-point parallel computing cluster based on general purpose graphics processing units (GPGPUs) achieved through parallel architecture of the GPGPU, which is a mini-supercomputer packed in a graphics card used for floating point operations Our major electron imaging simulation code is written in the CUDA programming language which uses a single-precision FFT routine in the CUFFT library We have been able to simulate inorganic structures of unprecedented complexity using this hardware These simulations were performed by Sonali Mitra a Ph.D student working under the supervision of Drs Vogt and Blom in the Department of Chemistry and Biochemistry at the University of South Carolina The work by Amit Singer and Yoel Shkolnisky (Chap 6) is a tour-de-force in explaining the mathematical theory cryo-transmission electron microscopy is based on What appears to many practitioners of electron microscopy as “black art” is deeply rooted in fundamental mathematics This chapter illustrates the deep-rooted connections between imaging and applied mathematics, illustrating what Eugene Wigner coined in 1960 as the “unreasonable effectiveness of mathematics in the natural sciences” (Communications on Pure and Applied Mathematics 13 (1): 1–14) We believe that the combination of state-of-the-art imaging using aberrationcorrected electron microscopy with applied and computational mathematics will enable a “new age” of imaging in both the hard and soft sciences This will leverage the huge infrastructure investments that have been made globally over the past 10 years in national laboratories, universities, and selected companies Tom Vogt would like to thank the Korean Ministry of Science, Education, and Technology for a Global Research Laboratory grant and the National Academies Keck Future Initiative for support We all would like to acknowledge the support from the Nanocenter, the Interdisciplinary Mathematics Institute, and the College of Arts and Sciences at the University of South Carolina for the realization of the above-mentioned workshops that helped shape our ideas presented in this volume Contents Kantianism at the Nano-scale Michael Dickson The Application of Scanning Transmission Electron Microscopy (STEM) to the Study of Nanoscale Systems N.D Browning, J.P Buban, M Chi, B Gipson, M Herrera, D.J Masiel, S Mehraeen, D.G Morgan, N.L Okamoto, Q.M Ramasse, B.W Reed, and H Stahlberg 11 High Resolution Exit Wave Restoration Sarah J Haigh and Angus I Kirkland 41 Compressed Sensing and Electron Microscopy Peter Binev, Wolfgang Dahmen, Ronald DeVore, Philipp Lamby, Daniel Savu, and Robert Sharpley 73 High-Quality Image Formation by Nonlocal Means Applied to High-Angle Annular Dark-Field Scanning Transmission Electron Microscopy (HAADF–STEM) 127 Peter Binev, Francisco Blanco-Silva, Douglas Blom, Wolfgang Dahmen, Philipp Lamby, Robert Sharpley, and Thomas Vogt Center of Mass Operators for Cryo-EM—Theory and Implementation 147 Amit Singer and Yoel Shkolnisky Index 179 ix GCAR—Theory and Implementation 167 Á Á 1 1 1 1 O O Qg1 !x ; !y D Pg1 !x ; !y e { x !x Cy !y / ; Á Á 2 2 2 2 O O Qg2 !x ; !y D Pg2 !x ; !y e { x !x Cy !y / : (18) O O Suppose that the common line between the (centered) projections Pg1 and Pg2 is 1 Og1 and r cos  ; r sin  / in Pg2 , with  and  , measured O r cos  ; r sin  / in P O O from the !x -axis in Pg1 and Pg2 , respectively Along the common line O O Pg1 r cos  ; r sin  D Pg2 r cos  ; r sin  ; (19) and so, O Qg1 r cos  ; r sin  e {r x cos  Cy sin  / O D Qg2 r cos  ; r sin  /e {r x cos  Cy sin  / ; (20) from which we get x cos  C y sin  x cos  y sin  D g1 ;g2 ; (21) where g1 ;g2 D O Qg1 r cos  ; r sin  1 arg : O r Qg2 r cos  ; r sin  / (22) Equation (21) is an equation for x , y , x , and y in terms of the known O O quantities Qg1 , Qg2 ,  , and  Given K projection images, there are 2K unknowns k k x ; y and K equations of the form of (21) Thus, we form the K 2K 2 system of linear equations given by (21), and solve it using least squares Note that this linear system is very sparse as each row contains only four non-zero elements The resulting matrix has a null space of dimension three (as demonstrated below), which reflects the fact that arbitrarily moving the origin of the object space induces another set of consistent translations x k ; y k in the projections, which also satisfy (21) Note that we need not use all K equations, but only equations that correspond to trustworthy common lines (e.g., equations that correspond to common lines that pass the voting procedure [17]) We can further filter the system of equations by choosing only equations that correspond to pairs for which the lefthand side in (21) is nearly constant for various values of r Although in theory the left-hand side of (21) should be constant for all r, this is not the case in practice due to discretization, noise, and measurement errors O Note that according to (19), if  and  are the common line between Pg1 and O Pg2 , that is, O O Pg1 r cos  ; r sin  D Pg2 r cos  ; r sin  ; 168 A Singer and Y Shkolnisky then also O Pg1 r cos  C O DPg2 ; r sin  C r cos  C ; r sin  C : To avoid this type of ambiguity, we restrict the angle  to range only from to , while the radius r goes from to In this setting, the angles of the common O O line satisfy Â1 < and Â2 < , and the rays in Pg1 and Pg2 corresponding to the common line can match in one of two orientations: either O O Pg1 r cos  ; r sin  D Pg2 r cos  ; r sin  (23) for all r, or, O O Pg1 r cos  ; r sin  D Pg2 r cos  ; r sin  (24) for all r Thus, to detect if some pair of rays is a common line, we need to test if the two rays have high similarity (see (23)), or if the flipped version of one ray has high similarity with the other (unflipped) ray (see (24)) This changes the form of the shift equations (21) as we now describe O O If the rays of the common line between Pg1 and Pg2 match in the same orientation of r, that is, if (23) holds, then by (20), we get (21) again, that is, x cos  C y sin  x cos  y sin  D g1 ;g2 ; (25) O O where g1 ;g2 is defined in (22) If the rays of the common line between Pg1 and Pg2 match in opposite orientations, that is, if (24) holds, then by (18) and (24) we get O Qg1 r cos  ; r sin  e {r x cos  Cy sin  / O D Qg2 r cos  ; r sin  /e {r x cos  Cy sin  / ; from which we get x cos  C y sin  C x cos  C y sin  D where g1 ;g2 g1 ;g2 ; (26) is again defined by (22) 5.2 Detecting Common Lines Forming the shift equations in the form of (25) and (26) requires to detect common lines between pairs of projections in the presence of unknown shifts We will assume that (23) holds The case when (24) holds is treated similarly As a result of (23), ˇ ˇ ˇ ˇ ˇ ˇO ˇ ˇO ˇPg1 r cos  ; r sin  /ˇ D ˇPg2 r cos  ; r sin  /ˇ ; GCAR—Theory and Implementation 169 and from (18), we get ˇ ˇ ˇ ˇ ˇ O ˇ ˇ O ˇ ˇQg1 r cos  ; r sin  /ˇ D ˇQg2 r cos  ; r sin  /ˇ : (27) Hence, to detect common lines between projections that were shifted by some unknown shift, in principle, it is possible to take the polar Fourier transform of each (shifted) projection, and find common lines between the absolute values of the Fourier rays In practice, detecting common lines by discarding the phases performs poorly, especially in the presence of high levels of noise, since the phases contain a significant portion of the information We therefore present an alternative approach, which does not involve discarding the phases To further understand the relation between two noncentered projections along their common line, we rewrite (20) in the form O Qg1 r cos  ; r sin  e {rs O D Qg2 r cos  ; r sin  ; (28) wheres D x cos  C y sin  x cos  y sin  From (28) and (22), we get that the one-dimensional shift s is exactly the phase factor g1 ;g2 Equation (28) also shows that the one-dimensional inverse Fourier transforms of O O Qg1 r cos  ; r sin  / and Qg2 r cos  ; r sin  / (with respect to r) are related by a one-dimensional shift of s pixels The one-dimensional inverse Fourier transform O of Qg1 r cos  ; r sin  / is the one-dimensional profile obtained by projecting the two-dimensional image Qg1 x; y/ in the direction perpendicular to  , namely, computing its two-dimensional Radon transform in the direction perpendicular to O the common line A similar claim holds for Qg2 r cos  ; r sin  / Due to the observation that g1 ;g2 D s, the shift equation (25) takes the form x cos  C y sin  x cos  y sin  D s; (29) if both rays of the common line have the same orientation Similarly, (26) takes the form x cos  C y sin  C x cos  C y sin  D s; (30) if the rays have opposite orientations In both cases, s is the one-dimensional shift O O between the Fourier rays of the common line between Pg1 and Pg2 Following (28), we detect the common line between projections Qg1 x; y/ and Qg2 x; y/ using the following procedure: O O Compute the polar Fourier transforms Qg1 and Qg2 Select a maximal l allowed shift smax and a shift resolution ıs (e.g., one or two m pixels) Set M D smax ıs Q O For all shifts sDmıs , mD M; : : : ; M , compute Qg1 r; Â/DQg1 r; Â/e {rs Q g1 is the polar Fourier transform of Qg1 x; y/, where each of the Fourier rays Q has been shifted by a one-dimensional shift s Denote the common line between 170 A Singer and Y Shkolnisky Q O Qg1 and Qg2 by cg1 ;g2 m/, the corresponding correlation coefficient between the lines by g1 ;g2 m/, and the current relative shift s by ıg1 ;g2 m/ Note that it is required to check both the case where both rays have the same orientation, and the case where they have opposite orientations Define the common line between Qg1 x; y/ and Qg2 x; y/ by cg1 ;g2 m0 /, where m0 D arg max m g1 ;g2 m/: Using the angles determined by cg1 ;g2 m0 / and the relative shift ıg1 ;g2 m0 /, construct the shift equations (29) and (30) This procedure basically shifts any pair of rays in any two Fourier-transformed projections by a relative one-dimensional shift of s D mıs ; m D M; : : : ; M , and defines the common line as the pair whose similarity after the one-dimensional shift (and possibly a flip) is maximal The one-dimensional relative shift ıg1 ;g2 m0 / Q O between this pair of rays, together with their angles  and  in Qg1 and Qg2 , specified by cg1 ;g2 m0 /, are used to form the shift equations (29) and (30) 5.3 Measuring the Shift Estimation Error 2K matrix, denoted T , The system of shift equations can be written using a K where each row corresponds to the common line between some two projections Pgk1 and Pgk2 Each row in T has the form (assuming, say, that both Fourier rays match in the same orientation) cos  k1 sin  k1 cos  k2 sin  k2 This row has only four non-zero entries, at the columns that correspond to x k1 , y k1 , x k2 , and y k2 Any solution to this system, denoted in vector form by T s D x ; y ; : : : ; x K ; y K , can be written as s D r C V q, where V is a basis for the null space of T This null space has dimension three The reason for the null space is that the three-dimensional volume is determined up to an O(3) transformation and a shift of the origin of the coordinate system When moving the origin of the three-dimensional volume, each Fourier ray in each of its projections is also shifted (some phases appear along each ray) The common line between any two projections is shifted by exactly the same amount in each of the two projections, since both rays of the common line correspond to the same three-dimensional ray Thus, each position of the three-dimensional origin induces a two-dimensional shift into each projection, which preserves the relative shift along each common line This means that for each location of the three-dimensional origin we get the same system of shift equations However, each three-dimensional shift of the origin corresponds to a different set of two-dimensional shifts Thus, we have three degrees of freedom in specifying the two-dimensional shifts, and so the dimension of the solution’s space (the two-dimensional shifts) is three GCAR—Theory and Implementation 171 If s1 D r1 C V q1 and s2 D r2 C V q2 are two solutions to the shift equations, then the accuracy of the shift estimation is given by kr1 r2 k=kr1 k, or equivalently, kPV ? s1 s2 /k=kPV ? s1 k, where PV ? is the orthogonal projection on the subspace perpendicular to V If T is the matrix that corresponds to the shift equations, and T D US V1T is the SVD of T , then the null space of T is given by the columns of V that correspond to zero singular values, that is, V=V1 (:,end-2:end) In practice, the three smallest singular values are not zero but rather much smaller than the others 5.4 Examples The center determination algorithm is demonstrated in Fig 14 We generated K D 200 noiseless projections of the E coli 50S ribosomal subunit, corresponding to random uniform rotations in SO(3) Each projection is centered and of size 129 129 We then shifted each of the K D 200 projections randomly in the x and y directions by either 0, 5, or 10 pixels The input to our algorithm is the set of K D 200 shifted projections We used noiseless projections to demonstrate that any artifacts in the reconstruction are not due to noise nor misidentification of common lines, but rather due to ignoring the relative shifts between projections We computed the polar Fourier transform of each projection using L D 72 For each pair of images, we shifted all rays in one of the images by s, correlated all rays in one image with all rays in the second image, and took the pair with maximum correlation (over all values of s) as the common line Since common lines between two images can match in one of two orientations, when searching for common lines, we took into account the two cases (as explained in Sect 5.2) In the p implementation used to generate Fig 14, we used shifts s between 20 pixels p and 20 pixels, with steps of 0.5 pixels In practice, there is no reason to use such a fine sampling of the translations Figure 14a shows the spectrum of the operator A from (12), constructed from the common lines detected in the presence of shifts The multiplicities of 1, 3, indicate that common lines were correctly detected Figure 14b is the histogram of the angle estimation error, namely, the histogram of the angle between the estimated and the true orientation of each Fourier ray Since common lines were correctly detected, the estimation errors are very small (less than 0:5ı ) Next, using the common line information and the phase shift between the Fourier rays corresponding to each common line, we constructed the shift equations (29) and (30) and, using least-squares, estimated the two-dimensional shift in each projection Figure 14c is a bar plot of the relative translation errors (200 translations in the x direction and 200 translations in the y direction) Figure 14d shows the first few singular values of the matrix that corresponds to the coefficients of the shift equations (T in Sect 5.3) A null space of dimension three is apparent, as predicted above Finally, we used the estimated orientations and translations to reconstruct the molecule Figure 14e is a three-dimensional rendering of the reconstructed 172 A Singer and Y Shkolnisky Fig 14 The effect of shifts in the projection images (a)–(f) were generated using K D 200 noiseless projections with L D 72 (a) Spectrum of the operator A constructed from the shifted projections while taking shifts into account (b) Histogram of the angle (in degrees) between the true direction ˇk;l S of each Fourier ray and its estimated direction (c) Bar plot of the error in estimating the shifts x-axis contains 400 points of the form x k ; y k /, k D 1; : : : ; 200 y-axis is the translation error measured as described in Sect 5.3 (d) Twenty smallest singular values of the matrix corresponding to the shift equations A null space of dimension three is apparent (e) Volume reconstructed from the shifted projections and their estimated orientations after centering each projection by its estimated translation (f) Volume reconstructed from the shifted projections and the estimated orientations, without correcting for the shifts volume, after centering each projection according to its estimated translation This structure is comparable to Fig On the other hand, in Fig 14f, we reconstructed the volume using the estimated orientations, but without centering the projections That is, the input for the reconstruction is the shifted projections and their correct orientations The artifacts in the reconstruction are clear, demonstrating the importance of accurate shift estimation and correction The artifacts are only due to the shifts, since the orientations are correctly estimated, and as Fig 14e demonstrates, these artifacts disappear once the projections are appropriately centered Reconstruction from Real Data In this section, we demonstrate that the algorithms of the previous sections allow to obtain reconstructions from real electron microscope data GCAR—Theory and Implementation 173 Fig 15 Raw projections from a dataset of 27,121 projection images of the E coli 50S ribosomal subunit Fig 16 Four class averages out of the 1,500 class averages of the experimental dataset A set of micrographs of E coli 50S ribosomal subunits was provided by M van Heel (this is the molecule whose clean model was used to generate simulated projections in previous sections) These images were acquired with a Philips CM20 ˚ at defocus values between 1.37 m and 2.06 m; they were scanned at 3.36 A/pixel, and particles were picked using the automated particle picking algorithm in EMAN Boxer All subsequent image processing was performed with the IMAGIC software package [18, 19] The particle images were phase-flipped to remove the phase ˚ ˚ reversals in the CTF and band-pass filtered at 1/150 A and 1/8.4 A The variancenormalized images were translationally aligned with the rotationally-averaged total sum The output of this preprocessing procedure was a dataset of 27,121 particles (projections images) Four raw projections from this dataset are shown in Fig 15 Without rotational alignment, the 27,121 particle images were classified using the MSA function into 1,500 classes Since the input images were translationally aligned, each of the 1,500 class averages is roughly centered We therefore mask each image using a (radially smoothly decaying) circular mask, which removes noise samples from each image and improves its SNR The masked class averages were used as the input for the GCAR algorithm Four class averages out of the 1,500 that were used as the input are shown in Fig 16 After finding all common lines between all pairs of images, we removed all common lines that were suspected to be wrong This was done using the voting algorithm described in [17] We then constructed the operator A (see (12)) using the filtered list of common lines, and used the iterative refinement described in Sect to estimate the orientations The output of the GCAR algorithm is illustrated in Figs 17 and 18 Figure 17a shows the spectrum of the operator A constructed 174 A Singer and Y Shkolnisky a c b 180 0.35 0.5 160 120 0.3 140 0.25 100 -0.5 0.2 80 0.15 -1 0.1 0.05 0 60 Spectrum (1) d 0 -1 -1 10 11 12 13 14 40 20 0.2 0.4 0.6 0.8 1.2 1.4 1.6 1.8 x 10 Common lines discrepancy (1) Embedding (1) e f 12 0.25 0.5 10 0.2 0.15 -0.5 0.1 -1 0 0.05 10 11 12 13 14 Spectrum (7) -1 -1 Embedding (7) 0 2000 4000 6000 8000 10000 12000 1400016000 Common lines discrepancy (7) Fig 17 Reconstruction from real data Fig 18 Three-dimensional rendering of the volume reconstructed from real data using the experimental data Figure 17b shows the three-dimensional embedding, as estimated by the algorithm, of the Fourier rays that correspond to the first 10 projections The threefold multiplicity in Fig 17a and the embedding in Fig 17b are distorted, due to misidentifications of common lines, that persist even after applying GCAR—Theory and Implementation 175 the filtering algorithm [17] The inconsistencies in the embedding are also evident in Fig 17c This figure was generated by taking each pair of Fourier rays that are supposed to correspond to a common line, and plotting the angle between their three-dimensional embeddings In Fig 17c, we show these angles sorted from small to large Ideally, two Fourier rays that correspond to a common line should have exactly the same embeddings, and so the angle between the embeddings should be zero However, due to errors, we see that this angle is large for many pairs of common lines These errors are manifested as distorted circles in Fig 17b To correct these errors, we applied the iterative refinement algorithm of Sect The output of the algorithm after seven iterations of refinement is shown in Figs 17d–f In Fig 17d, we see the spectrum of the operator A from (12) after seven refinement iterations, where the threefold multiplicity in the spectrum has been restored, implying that the errors in the common lines matrix have been removed during the refinement iterations The embedding in Fig 17e looks much cleaner compared to Fig 17b The plot of common line discrepancy in Fig 17f shows that each pair of common lines is now embedded into points that are at most 10ı apart In our implementation, we stopped the iterations once the maximal discrepancy between common line embeddings was below 10ı This was achieved after only seven refinement iterations Finally, we used the noisy input images and the estimated orientations to reconstruct the molecule A three-dimensional rendering of the reconstructed volume is shown in Fig 18 The volume was filtered using a Gaussian filter with D 0:7 The dataset used in this experiment is of the same molecule that was used to generate the simulated projections in Sects 3–5 It is evident that the reconstruction obtained from real projections in Fig 18 is consistent with the clean model of the same molecule shown in Fig Summary and Outlook GCAR is an algorithm for finding the viewing directions of cryo-EM projection images from their common lines In this chapter we reviewed the basic mathematical ideas underlying GCAR, discussed specific implementation details and demonstrated its performance on both simulated and real microscope data We are currently developing and researching other algorithms for three-dimensional structure determination of macromolecules from cryo-EM data A specific promising research direction that we would like to point out here is a semidefinite programming (SDP) approach for finding the viewing directions from the common lines Preliminary experimental results show that the SDP based algorithm can recover the viewing directions for even higher levels of noise At the same time, we are also developing spectral based algorithms for improving the preliminary class averaging stage We are confident that the combination of these newly developed methods will prove valuable in single particle reconstruction methodologies 176 A Singer and Y Shkolnisky Acknowledgments We would like to thank Fred Sigworth and Ronald Coifman for introducing us to the cryo-EM problem and for many stimulating discussions We also thank Tom Vogt and Wolfgang Dahmen for their hospitality at the Industrial Mathematics Institute and the NanoCenter at the University of South Carolina during “Imaging in Electron Microscopy 2009” The project described was supported by Award Number R01GM090200 from the National Institute of General Medical Sciences The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of General Medical Sciences or the National Institutes of Health Molecular graphics images were produced using the UCSF Chimera package from the Resource for Biocomputing, Visualization, and Informatics at the University of California, San Francisco (supported by NIH P41 RR-01081) References Frank J (2006) Three-dimensional electron microscopy of macromolecular assemblies: visualization of biological molecules in their native state Oxford van Heel M, Gowen B, Matadeen R, Orlova EV, Finn R, Pape T, Cohen D, Stark H, Schmidt R, Schatz M, Patwardhan A (2000) Single-particle electron cryo-microscopy: towards atomic resolution Q Rev Biophys 33(04):307–369 Wang L, Sigworth FJ (2006) Cryo-EM and single particles Physiology (Bethesda) 21:13–8 Review PMID: 16443818 [PubMed—indexed for MEDLINE] Henderson R (2004) Realizing the potential of electron cryo-microscopy Q Rev Biophys 37(1):3–13 Review PMID: 17390603 [PubMed—indexed for MEDLINE] Chiu W, Baker ML, Jiang W, Dougherty M, Schmid MF (2005) Electron cryomicroscopy of biological machines at subnanometer resolution Structure 13(3):363–372 Review PMID: 15766537 [PubMed—indexed for MEDLINE] Van Heel M (1987) Angular reconstitution: a posteriori assignment of projection directions for 3D reconstruction Ultramicroscopy 21(2):111–123 PMID: 12425301 [PubMed—indexed for MEDLINE] Coifman RR, Shkolnisky Y, Sigworth FJ, Singer A (2010) Reference free structure determination through eigenvectors of center of mass operators Appl Comput Harmonic Anal 28(3):296–312 Natterer F (2001) The mathematics of computerized tomography Classics in Applied Mathematics SIAM: Society for Industrial and Applied Mathematics Pettersen EF, Goddard TD, Huang CC, Couch GS, Greenblatt DM, Meng EC, Ferrin TE (2004) UCSF Chimera—a visualization system for exploratory research and analysis J Comput Chem 25(13):1605–1612 10 Natterer F, Wˆ bbeling F (2001) Mathematical methods in image reconstruction Monographs u on Mathematical Modeling and Computation SIAM: Society for Industrial and Applied Mathematics, First edition 11 Dutt A, Rokhlin V (1993) Fast Fourier transforms for nonequispaced data SIAM J Sci Comput 14(6):1368–1393 12 Greengard L, Lee J-Y (2004) Accelerating the nonuniform fast Fourier transform SIAM Rev 46(3):443–454 13 Beylkin G (1995) On the fast Fourier transform of functions with singularities Appl Comput Harmonic Anal 2:363–381 14 Potts D, Steidl G, Tasche M (2001) Fast Fourier transforms for nonequispaced data: a tutorial In: Benedetto JJ, Ferreira P (ed) Modern sampling theory: Mathematics and Applications (Birkhă user) a 15 Averbuch A, Shkolnisky Y (2003) 3D Fourier based discrete Radon transform Appl Comput Harmonic Anal 15(1):33–69 GCAR—Theory and Implementation 177 16 Averbuch A, Coifman RR, Donoho DL, Israeli M, Shkolnisky Y (2008) A framework for discrete integral transformations I—the pseudo-polar Fourier transform SIAM J Sci Comput 30(2):764–784 17 Singer A, Coifman RR, Sigworth FJ, Chester DW, Shkolnisky Y (2010) Detecting consistent common lines in cryo-EM by voting J Struct Biol 169(3):312–322 18 Stark H, Rodnina MV, Wieden HJ, Zemlin F, Wintermeyer W, van Heel M (2002) Ribosome interactions of aminoacyl-tRNA and elongation factor Tu in the codon-recognition complex Nature Struct Mol Biol 9:849–854 19 van Heel M, Harauz G, Orlova EV, Schmidt R, Schatz M (1996) A new generation of the IMAGIC image processing system J Struct Biol 116(1):17–24 Index A Aberration measurement, 42, 51–59, 67 Acquisition, 18, 51, 74, 75, 89–91, 93–95, 100, 102, 109, 112, 128, 130, 133, 135, 142 Additive noise, 87, 106, 109, 130 ADF See Annular dark-field imaging (ADF) Adjacency matrix of the graph, 155 Aggregated measurements, 99, 112 Angles estimation error, 160, 171 Annular dark-field imaging (ADF), 18, 92, 129 Atomic column, 14, 15, 17, 21, 25, 26, 91, 93–95, 97, 99–101, 103, 107, 110, 129 B Back projection, 144, 166 Band limited, 149, 154 Beam damage, 17, 20, 21, 97, 128, 131, 132, 135, 136, 144 Beam sensitive material, 18, 92, 96, 128, 131, 132, 144 Bernoulli matrices, 83, 85 Best approximation, 78 Bitstream, 74 Blurring, 18, 30, 31, 33, 86, 130, 134 Bounded variation (BV), 86–88 Bounding box, 165 Bregman-iteration, 88 Bump function, 94, 95, 98, 102, 105, 110 BV See Bounded variation (BV) C Canonical basis, 75 Center estimation, 164–172 Change of basis, 74, 75 Clathrate, 20–24, 26 Coherent, 14, 43, 45, 50, 52, 58, 99, 110 Common line, 148, 153–155, 158, 159, 161–163, 165–171, 173–175 “Common line” property, 153–155 Compressed sensing (CS), 73–124 matrix, 81 measurement, 98, 99 Compressed sensing pair (CS pair), 85 Compression, 75, 78 Compression scheme, 79 Concentration inequality, 83, 84 Concentration parameter , 105, 106 Convex optimization, 85, 88 Cryo-electron microscopy (cryo EM), 146–175 cryo EM See Cryo-electron microscopy (cryo EM) CS See Compressed sensing (CS) CS pair See Compressed sensing pair (CS pair) D 2D crystallography, 20 Decay rate, 80, 81 Decoder, 83–86, 88–90, 92 Denoising, 86–88, 97, 133–138, 141, 142 DFT See Discrete Fourier transform (DFT) Discrete Fourier basis, 77 Discrete Fourier transform (DFT), 89, 90, 158, 159 Discrete wavelet basis, 74, 75, 87 Dislocation cores, 27–29 Distance notions, 133, 135, 136 3D-structure, 2, 91, 112, 147–149, 175 Dwell time, 18, 19, 94, 130, 139 T Vogt et al (eds.), Modeling Nanoscale Imaging in Electron Microscopy, Nanostructure Science and Technology, DOI 10.1007/978-1-4614-2191-7, © Springer Science+Business Media, LLC 2012 179 180 E E coli 50S ribosomal subunit, 151, 152, 171, 173 Electron dose, 17, 18, 20, 21, 94, 96, 97, 99, 130 Electron tomography, 74, 91, 112–124 Encoder, Encoder–decoder pair, 82 Encoding scheme, 79 Enforced symmetry, 23 Environmental effects, 95 Environmental noise, 128 Exact reconstruction, 90, 118 Exit wave, 41–68 F Fourier projection-slice theorem, 152–155, 158, 159, 162, 164–166 Fourier rays, 159, 161–164, 169–171, 174, 175 Fourier shift property, 166 Fourier-slice-theorem, 115 Fourier transform, 18–22, 45, 47, 55, 58, 60, 61, 64, 89, 90, 115, 139, 152–154, 158–160, 162, 164–166, 169–171 Frozen phonon model, 100 Frozen phonon simulation, 105 G GaAsN quantum wells, 24 Gaussian blur, 32, 100 Gaussian filter, 161, 175 Gaussian normal distribution, 118 Gaussian white noise, 152 Globally consistent angular reconstruction (GCAR), 148, 152–163, 173, 175 Grain boundaries, 12, 13, 21, 27, 28, 30 Greedy algorithms, 86 H HAADF See High-angle annular dark field (HAADF) HAADF–STEM See High-angle annular dark field scanning transmission electron microscopy (HAADF–STEM) High-angle annular dark field (HAADF), 13, 31, 92 High-angle annular dark field scanning transmission electron microscopy (HAADF–STEM), 24, 31, 32, 92, 93, 112, 124, 127–144 Index High quality micrographs, 128 Histogram, 33, 34, 76, 160, 163, 171, 172 I Image reconstruction, 90, 107, 132, 134 Information content, 89, 95, 96 Instance optimal, 82, 83, 85, 88 Intensity distribution, 46, 47, 92, 94, 105, 107–109, 112, 133 Inverse DFT, 158–159 IQ, 22, 23 Iterative refinement, 163–165, 173, 175 Iterative reweighted least squares, 86 K Kaczmarz-Iteration, 116, 118, 119, 121 K-term approximation, 78, 79, 82, 84, 89 L Least squares distortion, 77 Least squares norm, 78, 79, 83, 85 Level of incoherence, 102 Linear approximation, Linear imaging approximation, 48, 60, 115 `1 minimization, 85, 86, 88 Logan–Shepp Phantom, 89, 116, 117, 120–123 Low dosage micrographs, 128 Low-dose imaging, 17–20 `1 penalization, 88 M Magnification, 18, 47, 55, 76, 94, 130, 139 MAL See Maximum likelihood algorithm (MAL) Maximum likelihood algorithm (MAL), 65 M1 catalyst, 76, 91, 92, 95, 128, 131, 132, 136–139 Measurement, 13, 42, 74, 128 Median averaging, 136, 139 Micrograph, 18, 92, 109, 111, 128–130, 132–143 MIMAP See Multiple input maximum a posteriori (MIMAP) Minimum variance method, 52 Missing wedge, 17, 114, 115, 118, 120 Mitsubishi catalyst, Monotonicity Property, 78 Multiple input maximum a posteriori (MIMAP), 65 Mutual-information-registration code, 135 Mutual similarity, 134 Index N Nanoparticle size distribution, 30–34 Nearest neighbor interpolation, 160 NESTA, 102, 105–111, 117 NLM See Nonlocal means (NLM) Non-Gaussian noise, 151 Nonlocal means (NLM), 127–144 O Orientation assignment algorithm, 148 Orientation refinement, 162–164 “Orientation revealing” graph, 155 “Orientation revealing” operator, 154–156 P Paraboloid method, 60–62 PCA See Principal component analysis (PCA) PCI See Phase contrast index (PCI) function PCTFs See Phase contrast transfer function (PCTFs) Penalty parameter, 102, 103, 106 Phase contrast index (PCI) function, 59, 61 Phase contrast transfer function (PCTFs), 50–51 Phase object approximation (POA), 49–50 POA See Phase object approximation (POA) Point defect analysis, 26 Poisson noise, 108 Polar Fourier transform, 154, 158, 169, 171 Positive semi-definite matrix, 157 Principal component analysis (PCA), 158, 159, 161, 163, 164 Projection image, 149–154, 158–159, 162, 164–167, 172, 173, 175 Q Quantization of coefficients, 79 R Random matrices, 83–85, 88 Ray integrals, 115, 118 Rearrangement property, 78 Reciprocal space, 22, 23, 43, 58, 95, 96 Reconstruction, 23, 60, 74, 89–91, 102, 104, 106–108, 115, 118–123, 132, 134, 148–151, 154, 161, 171–175 Recovery procedures, 102, 107 Registration of denoised frames, 135 Regularization, 85, 115, 116, 120, 142 181 Relaxation time, 91, 98 Resolution level, 74 Restoration filters linear, 62–64 non-linear, 60 Restricted isometry property (RIP), 84, 86, 88, 99, 117 Rotation group SO(3), 149 S Sample, 2, 13, 14, 17, 20, 21, 24, 33–35, 49, 52, 55, 56, 59, 61, 63, 64, 66, 74, 75, 81, 82, 84, 92–94, 102, 112, 128–130, 136, 137, 148, 150, 154, 158, 161, 163, 173 Sampling, 61, 65, 74, 75, 81–84, 94, 171 Scanning transmission electron microscope (STEM), 11–35, 73, 90, 92–101, 105, 109, 112, 124, 128–131, 136, 140, 142, 144 SDP See Semi-definite program (SDP) Self-similarities, 134, 135 Semi-definite program (SDP), 157, 175 Sensing matrices, 81, 110 Sensor noise, 77, 88 SESOP, 102–106 Shift equations, 166–168, 170–172 Shift estimation, 170–172 Signal-to-noise ratio (SNR), 20, 64, 74, 93, 97–99, 128, 130, 135, 151, 161–163, 173 Similarity check, 129 Simulation, 15, 24, 26, 34, 50, 92, 100, 105, 130, 164 Single-particle reconstruction (SPR) SNR See Signal-to-noise ratio (SNR) Soft thresholding, 87, 88 Sparse adjacency matrix, 155 representation, 77, 89, 95 sequence, 77 Sparsity, 75–78, 80–82, 85, 86, 88, 95–96, 98–100, 102, 103, 106, 110, 116, 124 Sparsity level, 78, 89, 96, 106 Spatial coherence, 45–47, 66 Spatial registration, 134 “spider,” 155, 156 Stability, 27, 28, 82, 103, 105 STEM See Scanning transmission electron microscope (STEM) Strontium titanite, 91, 92 Structure determination, 13, 148, 149, 175 Subsampling, 90, 130 182 Index Support, 3, 5, 14, 30–32, 58, 77, 85, 93, 100, 134, 148, 149 Synthetic image, 102, 117 U Unit sphere S , 153, 154, 157 Unmixing coordinates, 157–158, 160 T Temporal coherence, 45–47, 50 Three-dimensional electron microscopy, 147–148 Tilt angle, 112–114, 116, 118–123 Time series, 128–139, 141, 144 Tomographic reconstructions, 91 Tomography, 17, 74, 91, 149, 150, 152 Total Variation (TV), 86, 87, 90, 117, 119, 122, 123 minimization, 117 penalization, 117 reconstruction, 118, 120, 122, 123 Transport of Intensity, 66 TV See Total variation (TV) V Voting algorithm, 173 W Warping, 130, 131, 135 Wave aberration function, 42–45, 47, 51–53, 56, 59, 68 Wavelet representation, 76 Z Z-contrast, 12–25, 29, 30, 34 Zeolites, 91, 92, 95, 128, 139, 140, 144 ... Thomas Vogt • Wolfgang Dahmen • Peter Binev Editors Modeling Nanoscale Imaging in Electron Microscopy 123 Editors Thomas Vogt NanoCenter and Department of... rights Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com) Preface Imaging with electrons, in particular using scanning transmission electron microscopy. .. entitled ? ?Imaging in Electron Microscopy? ?? in 2009 and 2010 and “New Frontiers in Imaging and Sensing” in 2011 At these workshops world-class practitioners of electron microscopy, engineers, and