Radiometrically Accurate Thermal Vehicle Targets for Synthetic Vi

Rochester Institute of Technology RIT Scholar Works Theses 12-9-2016 Radiometrically Accurate Thermal Vehicle Targets for Synthetic Video Development David B Rhodes Follow this and additional works at: https://scholarworks.rit.edu/theses Recommended Citation Rhodes, David B., "Radiometrically Accurate Thermal Vehicle Targets for Synthetic Video Development" (2016) Thesis Rochester Institute of Technology Accessed from This Thesis is brought to you for free and open access by RIT Scholar Works It has been accepted for inclusion in Theses by an authorized administrator of RIT Scholar Works For more information, please contact ritscholarworks@rit.edu Radiometrically Accurate Thermal Vehicle Targets for Synthetic Video Development by David B Rhodes A.S Corning Community College, 2004 B.S SUNY Brockport, 2006 A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in the Chester F Carlson Center for Imaging Science of the College of Science Rochester Institute of Technology December 9, 2016 Signature of the Author Accepted by Coordinator, M.S Degree Program Date CHESTER F CARLSON CENTER FOR IMAGING SCIENCE ROCHESTER INSTITUTE OF TECHNOLOGY COLLEGE OF SCIENCE ROCHESTER, NEW YORK CERTIFICATE OF APPROVAL M.S DEGREE THESIS The M.S Degree Thesis of David B Rhodes has been examined and approved by the thesis committee as satisfactory for the thesis required for the M.S degree in Imaging Science Dr Zoran Ninkov, Thesis Advisor Dr J Daniel Newman Dr David Messinger Date ii Radiometrically Accurate Thermal Vehicle Targets for Synthetic Video Development by David B Rhodes Submitted to the Chester F Carlson Center for Imaging Science in partial fulfillment of the requirements for the Master of Science Degree at the Rochester Institute of Technology Abstract Collecting large scientific quality thermal infrared image and video data sets is an expensive time consuming endeavor Thermal infrared imagers cost much more than comparable visible systems and require skilled experienced operators Also, time and experienced personnel are required to collect quality ground truth Often it is advantageous to perform computer simulations as an alternative to collecting image and video data with real camera systems As long as enough physics is incorporated into the models to give accurately comparable results to real imagery, simulated data can be used interchangeably Generating synthetic images and video has the added benefit of being flexible as the user has control over every aspect of the simulation Simulations are not subject to restrictions such as location, weather conditions, time of day, or time of year Ground truth is assigned instead of measured in the synthetic world so it is known a priori This thesis illustrates a method of using the Digital Image and Remote Sensing Image Generation (DIRSIG) software to create simulated infrared images and video of validated thermal target vehicle models inside thermal infrared wide-area scenes A finite difference heat propagation and surface temperature solver, ThermoAnalytics Multi-Service Electro-optic Signature (MuSES iii TM ), iv was used to accurately model the emissive thermal target vehicles Validation of the thermal target vehicle model was performed using images taken from a laboratory calibrated MWIR camera Images taken with the calibrated camera of the same type of vehicle as the target model were compared to the synthetic images for the same conditions for validation Target vehicle motion was added to the simulations through the use of Simulation of Urban Mobility (SUMO), DIRSIGs movement files, and custom python scripting The output images from DIRSIG were then laced together into video The resulting video was used to test three tracking algorithms illuminating each one’s strengths and weaknesses Acknowledgements I would like to thank my thesis committee members: Dr Zoran Ninkov, Dr Dan Newman, and Dr Dave Messinger My research has been guided by your support and guidance Thank you Zoran for getting me involved with DIRSIG while I was still at U of R Thanks to CEIS and ITT (now Harris Corporation) for funding my research Without funding I would not have been able to finish Thank you Dan Newman, Paul Lee, and Greg Gosian for offering suggestions on how my research should proceed from a programmatic point of view Thank you Ross Robinson, Kenny Fourspring, and Kyle Ausfeld for being in the lab to bounce questions off and helping with my numerous data collections Thanks you Judy Pipher and Craig McMurtry from U of R for helping me to understand infrared detector arrays and offering your camera system and truck for data collections Thank you Rolando Raque˜ no, Andy Scott, Scott Brown, Adam Goodenough, Niek Sanders, Mike Gartley, and Dave Pogorzala for answering my many DIRSIG questions Thank you Mike Presnar for helping me get a good start on making video in DIRSIG Thank you to Jason Faulring for helping me with the digital interface to the KIR-310 camera and operating WASP and WASP Lite on data collections Thank you Nina Raque˜ no for loaning me temperature measurement equipment and measuring materials for me Thank you Dr Schott for teaching me about remote sensing and for digging into the vault for old THERM validation documentation that I’m sure not exist anywhere else Thank you Brett Matzke, Jared Herweg, and Kenny Fourspring for keeping me sane The many bike rides and times we out kept me from imploding under the stress of being a single parent and a full-time graduate student Thank you Diane Alsup for proof reading this thesis and spending your life with me You are all very much appreciated v This thesis is dedicated to my four children, Patrick, Anthony, Debbie, and Kylie Your love and support has kept me going these many years I am very proud of the fine people you have grown to become vi Curriculum Vitae The author was born on September 23, 1974 in Elmira, NY to Gerald and Regina Rhodes He attended Corning Community College part-time from 2000 to 2004, and graduated highest program GPA with an Associates in Science He attended SUNY Brockport from 2004 to 2006 graduating Cum Laude with a Bachelor of Science dual degree in Mathematics and Physics He attended University of Rochester from 2006 to 2009 completing the Physics Coursework before leaving to attend RIT from 2009 to 2012 The author from 2012 to the time of the publishing of this thesis is employed as a Senior Image Scientist working for Harris Corporation, Rochester NY vii Contents Acknowledgements v Dedication vi Curriculum Vitae vii Table of Contents x List of Figures xiv List of Tables xiv List of Symbols xv Introduction 1.1 Motivation 1.2 Previous Work and Existing Methods 1.3 Approach Background Information 2.1 10 Introduction 10 viii ix CONTENTS 2.2 Radiometry 10 2.2.1 Introduction 10 2.2.2 Radiometric Quantities 11 2.2.3 Blackbody Radiation and Emissivity 13 2.3 Atmospheric Transmission 14 2.4 Heat Transfer 16 2.5 Remote Sensing 17 Software 3.1 20 DIRSIG 20 3.1.1 Introduction 20 3.1.2 Scene Geometry 23 3.1.3 Materials 25 3.1.4 Emissivity 26 3.1.5 Extinction 27 3.1.6 Atmosphere 28 3.1.7 Platform 28 3.1.8 Platform Motion 29 3.1.9 Tasks 30 3.1.10 DIRSIG’s Radiometry Solvers 30 3.1.11 DIRSIG’s Thermal Solver (THERM) 30 3.1.12 External Thermal Model (ThermoAnalytics MuSES TM ) 31 3.2 MODTRAN 31 3.3 MuSES 3.4 SUMO 33 TM 32 APPENDIX B MOVIE MAKER PYTHON CODE scaleImages.py contains all the functions used to properly scale and stretch DIRSIG images in order to produce a movie from them with makeMovie.py ’’’ import sys import os import numpy import pylab def readheader(header): ’’’Reads in ENVI header as a python dictionary’’’ headinfo = {} headerin = open(header, ’r’) head = headerin.readlines() for i in range(len(head)): if head[i].find(’=’) != -1: if head[i].find(’{’) != -1: if head[i].find(’}’) != -1: headinfo[head[i].split(’=’)[0].strip()] = head[i].split(’=’)[1].strip() pass elif head[i].find(’}’) == -1: headinfo[head[i].split(’=’)[0].strip()] 91 APPENDIX B MOVIE MAKER PYTHON CODE 92 = head[i].split(’=’)[1].strip() j = i while j != ’bracket found’: j += if head[j].find(’}’) == -1: headinfo[head[i].split(’=’)[0].strip()] = headinfo[head[i].split(’=’)[0].strip()] + head[j].strip() else: headinfo[head[i].split(’=’)[0].strip()] = headinfo[head[i].split(’=’)[0].strip()] + head[j].strip() j = ’bracket found’ else: headinfo[head[i].split(’=’)[0].strip()] = head[i].split(’=’)[1].strip() for i in range(len(headinfo.values())): if headinfo.values()[i].find(’{’) != -1: headinfo[headinfo.keys()[i]] = headinfo.values()[i].replace(’{’, ’’) headinfo[headinfo.keys()[i]] = headinfo.values()[i].replace(’}’, ’’) for i in range(len(headinfo.values())): if headinfo.values()[i].find(’,’) != -1: headinfo[headinfo.keys()[i]] = headinfo.values()[i].split(’,’) return headinfo def linearStretch(array): ’’’Simple linear pixel stretch’’’ APPENDIX B MOVIE MAKER PYTHON CODE stretchedArray = ((array - numpy.min(array))/(numpy.max(array) - numpy.min(array))) * (65535) stretchedArray = stretchedArray.astype(numpy.uint16) return stretchedArray, (numpy.min(array), numpy.max(array)) def clippedLinearStretch(array, percent, cols, rows): ’’’Simple linear pixel stretch with percent clip from lower and upper ends’’’ p = percent * 01 flatArray = array.flatten() (n, bins) = numpy.histogram(flatArray, bins=65335) cum = numpy.cumsum(n) keep = [] for i in range(len(cum)): if (cum[i] > cols * rows * p) and (cum[i] < (cols * rows) - (cols * rows * p)): keep.append(i) Min = bins[keep[0]] Max = bins[keep[-1]] for i in range(len(flatArray)): if flatArray[i] < Min: flatArray[i] = Min elif flatArray[i] > Max: flatArray[i] = Max newarray = flatArray.reshape(array.shape) 93 APPENDIX B MOVIE MAKER PYTHON CODE 94 stretchedArray = ((newarray - Min)/(Max - Min)) * (65535) stretchedArray = stretchedArray.astype(numpy.uint16) return stretchedArray, (Min, Max) def interactiveStretch(array, Min, Max): ’’’Simple linear pixel stretch with user defined minimum and maximum’’’ flatArray = array.flatten() for i in range(len(flatArray)): if flatArray[i] < Min: flatArray[i] = Min elif flatArray[i] > Max: flatArray[i] = Max array = flatArray.reshape(array.shape) stretchedArray = ((array - Min)/(Max - Min)) * (65535) stretchedArray = stretchedArray.astype(numpy.uint16) return stretchedArray, (Min, Max) def writeOutHist(bands, bandnames, arrays, path): for i in range(bands): fhand = open(path + bandnames[i] + ’_hist.txt’, ’w’) (n, bins) = numpy.histogram(arrays[i], bins=65335) for j in range(len(n)): fhand.write(str(n[j]) + ’ ’ + str(bins[j]) + ’\n’) fhand.close() APPENDIX B MOVIE MAKER PYTHON CODE 95 - ’’’ imageHist.py is a simple python program that plots the histogram of an image in a numpy array ’’’ import sys import numpy import pylab def plotHist(image, bandname): (n, bins) = image pylab.plot(bins, n) pylab.title(bandname) if name == ’ main ’: args = [] for arg in sys.argv: args.append(arg) if len(args) == 4: path = args[1] bands = eval(args[2]) APPENDIX B MOVIE MAKER PYTHON CODE 96 bandnames = list(eval(args[3])) hists, data = [], [] for i in range(bands): fhand = open(path + bandnames[i] + ’_hist.txt’, ’r’) hists.append(fhand.readlines()) for i in range(len(hists)): n, bins = [], [] for j in range(len(hists[i])): n.append(hists[i][j].split()[0]) bins.append(hists[i][j].split()[1]) data.append([n, bins]) for i in range(bands): pylab.figure(i+1) (n, bins) = data[i] pylab.plot(bins, n) pylab.title(bandnames[i]) pylab.show() else: print ’python imageHist.py ’ - ’’’ DIRSIG2PGM.py is a python program that converts DIRSIG ENVI images to APPENDIX B MOVIE MAKER PYTHON CODE 97 PGM image using numpy arrays Recall PGM uses big endian and ENVI uses the system byte order (Intel is little endian) DIRSIG output images are 64-bit double precision floating point numbers with band interleaved by pixel PGM images are or 16-bit integer numbers with band interleaved by pixel For now Rows and Columns are hardcoded! Syntax: python DIRSIG2PGM.py ’’’ import numpy import sys, os def readin(infile): filein = open(infile, ’r’) array = numpy.fromfile(file = filein, dtype = numpy.float64) return array def separateBands(array): APPENDIX B MOVIE MAKER PYTHON CODE ’’’numpy arrays are (bands, rows, cols)’’’ print array.shape array = array.reshape(256, 256, 2) MWArray = array[:,:,0] LWArray = array[:,:,1] return MWArray, numpy.min(MWArray), numpy.max(MWArray), LWArray, numpy.min(LWArray), numpy.max(LWArray) def rescale(array, min, max): flatarray = array.flatten() for i in range(len(array)): if flatarray[i] < min: flatarray[i] = elif flatarray[i] > max: flatarray[i] = max array = flatarray.reshape(array.shape) scaledArray = ((array-min)/(max-min))*numpy.float64(2**16-1) scaledArray = scaledArray.astype(numpy.uint16) return scaledArray def swapOrder(array): swappedArray = array.byteswap() return swappedArray def PGM1out(file, array, cols, rows): 98 APPENDIX B MOVIE MAKER PYTHON CODE outfile = open(file, ’w’) outfile.write(’P5\n’) outfile.write(str(cols) + ’ ’ + str(rows) +’\n’) outfile.write(’65535\n’) outfile.write(array) outfile.close() def PGM2out(file, array, cols, rows): outfile = open(file, ’w’) outfile.write(’P5\n’) outfile.write(str(cols) + ’ ’ + str(rows) +’\n’) outfile.write(’65535\n’) outfile.write(array) outfile.close() args = [] for arg in sys.argv: args.append(arg) if len(args)==2: directory = args[1] # for walk through files for subdir, dirs, files in os.walk(directory): ’’’First we need to get a and max for the first frame’’’ firstFrameMinMax = [] for file in files: 99 APPENDIX B MOVIE MAKER PYTHON CODE 100 firstfile = readin(file) if file[-4:len(file)] == ’.img’: firstFrameMinMax.append((separateBands(firstfile))) MWMin = firstFrameMinMax[0][1] MWMax = firstFrameMinMax[0][2] LWMin = firstFrameMinMax[0][4] LWMax = firstFrameMinMax[0][5] print MWMin, MWMax, LWMin, LWMax ’’’Next we convert to PGM using the and max from the first frame’’’ for file in files: if file[-4:len(file)] == ’.img’: infile = file pgmfile1 = infile[0:-4]+’_LW.pgm’ pgmfile2 = infile[0:-4]+’_MW.pgm’ cols = 256 rows = 256 array = readin(infile) MWArray, MWmin, MWmax, LWArray, LWmin, LWmax = separateBands(array) scaledMWarray = rescale(MWArray, MWMin, MWMax) scaledLWarray = rescale(LWArray, LWMin, LWMax) swappedMWarray = swapOrder(scaledMWarray) APPENDIX B MOVIE MAKER PYTHON CODE swappedLWarray = swapOrder(scaledLWarray) PGM1out(pgmfile1, swappedMWarray, cols, rows) PGM2out(pgmfile2, swappedLWarray, cols, rows) 101 Bibliography [1] D Rhodes, Z Ninkov, J L Pipher, C W McMurtry, J D Newman, P P K Lee, G J Gosian, and M D Presnar, “Synthetic scene building for testing thermal signature tracking algorithms,” pp 781309–781309–11, 2010 [2] N Pinel, C Bourlier, and I Sergievskaya, “Thermal IR emissivity of oil films on sea surfaces under moderate winds,” in General Assembly and Scientific Symposium, 2011 XXXth URSI, pp 1–3, Aug 2011 [3] “SENSIAC ATR Algorithm Development Image www.sensiac.org/external/products/list_databases.jdf Database.” [4] V Venkataraman, G Fan, X Zhang, W Liu, and P Havlicek, J, “Automated target tracking and recognition using coupled view and identity manifolds for shape representation,” EURASIP Journal of Advances in Signal Processing , 2011 [5] R T Collins, X Zhou, and S K Teh, “An Open Source Tracking Testbed and Evaluation Web Site,” in IEEE International Workshop on Performance Evaluation of Tracking and Surveillance (PETS 2005), Jan 2005 [6] “Video Verification of Identity (VIVID) Tracking Evaluation.” www.vision.cse.psu.edu/data/vividEval/datasets/datasets.html [7] H Ling, L Bai, E Blasch, and X Mei, “Robust Infrared Vehicle Tracking across Target Pose Change using L1 Regularization,” in Information Fusion(FUSION), 2010 13th Conference on, pp 1–8, july 2010 [8] P Norton, “HgCdTe infrared detectors,” Opto-Electronics Review , pp 159–174, Sept 2002 [9] Pipher, J L et al, “InSb array for IRAC (InfraRed Array Camera) on SIRTF (Space InfraRed Telescope Facility),” in SPIE, 2000 102 BIBLIOGRAPHY 103 [10] J R Janesick, Scientific Charge-Coupled Devices, SPIE Press, Bellingham, Washington, 2001 [11] J Janesick and G Putnam, “Developments and Applications of High-Performance CCD and CMOS Imaging Arrays,” Annual Review of Nuclear and Particle Science 53, pp 253–300, 2003 [12] M G Gartley, Polarimetric Modeling of Remotely Sensed Scenes in the Thermal Infrared PhD thesis, Chester F Carlson Center for Imaging Science, Rochester Institute of Technology, 2007 [13] J Savage, C Coker, B Thai, O Aboutalib, A Chow, N Yamaoka, and C Kim, “Irma 5.2 multi-sensor signature prediction model,” 2007 [14] B M Flusche, An Analysis of Multimodal Sensor Fusion for Target Detection in an Urban Environment PhD thesis, Chester F Carlson Center for Imaging Science, Rochester Institute of Technology, 2011 [15] C Harvey, J Wood, P Randall, G Watson, and G Smith, “Simulation of a new 3d imaging sensor for identifying difficult military targets,” 2008 [16] A A Mitchell, J M Brewster, A W Haynes, and P A Richardson, “Modelling reflections from exact surfaces in cameosim,” pp 654304–654304–12, 2007 [17] F Grum, Optical Radiation Measurements Volume Radiometry, Academic Press, New York, NY, 1979 [18] G H Rieke, Detection of Light From the Ultraviolet to the Submillimeter, Cambridge University Press, Cambridge, UK, 2003 [19] W Wolfe, The Infrared Handbook, Office of Naval Research, Department of the Navy, Washington, DC, 1978 [20] E Dereniak and G Boreman, Infrared Detectors and Systems, John Wiley & Sons, Inc., New York, NY, 1996 [21] J R Schott, Remote Sensing The Image Chain Approach, Oxford University Press, New York, NY, 2007 [22] U.S Standard Atmosphere, 1976, National Oceanic and Atmospheric Administration, National Aeronautics and Space Administration, and United States Air Force, Washington, D.C., Oct 1976 104 BIBLIOGRAPHY [23] “Digital Image and Remote Sensing Image Generation (DIRSIG).” www.dirsig.org [24] A B Berk, L S Bernstein, and D C Robertson, “MODTRAN: A moderate resolution model for LOWTRAN 7,” Spectral Science 1022, p 48, Apr 1989 [25] “ThermoAnalytics MuSES Infrared Signature www.thermoanalytics.com/products/muses/ Prediction Software.” [26] K.-J Bathe, Finite Element Procedures, Prentice-Hall, New Jersey, 1996 [27] M Behrisch, L Bieker, J Erdmann, and D Krajzewicz, “Sumo - simulation of urban mobility: An overview,” in SIMUL 2011, The Third International Conference on Advances in System Simulation, pp 63–68, (Barcelona, Spain), Oct 2011 [28] A Baldridge, S Hook, C Grove, and G Rivera, “The aster spectral library version 2.0,” Remote Sensing of Environment 113(4), pp 711 – 715, 2009 [29] “MODIS UCSB Emissivity Library.” www.icess.ucsb.edu/modis/EMIS/html/em.html [30] B.-H Tang, H Wu, C Li, and Z.-L Li, “Estimation of broadband surface emissivity from narrowband emissivities,” Opt Express 19, pp 185–192, Jan 2011 [31] “The 3D Studio.” www.the3dstudio.com [32] “FFmpeg.” www.ffmpeg.org [33] “Netpbm.” www.netpbm.sourceforge.net [34] “ImageMagick.” www.imagemagick.org [35] “Information technology–Digital compression and coding of continuous-tone still images: Requirements and guidelines,” 1994 ISO/IEC 10918-1:1994 [36] “Information technology–Coding of audio-visual objects–Part 14: MP4 file format,” 2003 ISO/IEC 14496-14:2003 [37] J Herweg, J Kerekes, O Weatherbee, D Messenger, J van Aardt, E Ientilucci, Z Ninkov, J Faulring, N Raqueno, and J Meola, “Spectir hyperspectral airborne rochester experiment data collection campaign,” in SPIE Defense Security and Sensing 2012, Apr 2012 [38] D B Rhodes, Z Ninkov, J D Newman, P P K Lee, and G J Gosian, “Development of radiometrically accurate synthetic thermal infrared video for tracking algorithm evaluation,” pp 840306–840306–9, 2012 BIBLIOGRAPHY 105 [39] K Ausfeld, “Tracking of various targets in the infrared and issues encountered,” Master’s thesis, Chester F Carlson Center for Imaging Science, Rochester Institute of Technology, 2012 ... one of the VIVID videos is shown in Figure 1.2 Figure 1.2: Frame of DARPA VIVID video of vehicle moving through intersection There are also some efforts to simulate physically accurate thermal infrared... 63 B Movie Maker Python Code 72 Bibliography 102 List of Figures 1.1 Frame of SENSIAC ATR LWIR video of vehicle moving across field of view 1.2 Frame of DARPA VIVID video of vehicle moving through... satisfactory for the thesis required for the M.S degree in Imaging Science Dr Zoran Ninkov, Thesis Advisor Dr J Daniel Newman Dr David Messinger Date ii Radiometrically Accurate Thermal Vehicle Targets for