Introduction to Media Computation - A Multimedia Cookbook in Python (2002)

Introduction to Media Computation: A Multimedia Cookbook in Python Mark Guzdial December 16, 2002 ii Contents I Introduction Introduction to Media Computation 1.1 What is computer science about? 1.2 What Computers Understand 1.3 Media Computation: Why digitize media? 1.4 Computer Science for Non-Computer Scientists 1.4.1 It’s about communication 1.4.2 It’s about process Introduction to Programming 2.1 Programming is about Naming 2.1.1 Files and their Names 2.2 Programming in Python 2.2.1 Programming in JES 2.2.2 Media Computation in JES 2.2.3 Making a Recipe II 7 11 14 15 16 16 19 19 21 22 23 23 32 Sounds 43 Encoding and Manipulating Sounds 3.1 How Sound is Encoded 3.1.1 The Physics of Sound 3.1.2 Encoding the Sound 3.1.3 Using MediaTools for looking at captured sounds 3.2 Manipulating sounds 3.2.1 Open sounds and manipulating samples 3.2.2 Introducing the loop 3.2.3 Changing the volume of sounds iii 45 45 45 52 58 60 60 63 67 CONTENTS iv 3.2.4 3.2.5 3.2.6 3.2.7 Manipulating different sections of the sound Splicing sounds Backwards sounds Changing the frequency of sounds differently 77 81 85 87 Creating Sounds 97 4.1 Creating an Echo 98 4.1.1 Creating Multiple Echoes 99 4.1.2 Additive Synthesis 100 III Pictures 113 Encoding and Manipulating Pictures 5.1 How Pictures are Encoded 5.2 Manipulating Pictures 5.2.1 Exploring pictures 5.2.2 Changing color values 5.2.3 Copying pixels 5.2.4 Replacing Colors 5.2.5 Combining pixels 5.3 Color Figures 115 115 119 126 127 134 145 152 159 Creating Pictures 165 6.1 Drawing on images with pixels 165 6.2 Drawing with drawing commands 166 IV Meta-Issues: How we what we Design and Debugging 7.1 Designing programs 7.1.1 Top-down 7.1.2 Bottom-up 7.2 Techniques of Debugging V Files 169 171 171 171 171 172 175 Encoding, Creating, and Manipulating Files 177 8.1 How to walk through a directory 177 CONTENTS 8.2 VI v Copying files 178 Text 181 Encoding and Manipulation of Text 183 9.1 A recipe to generate HTML 183 9.2 Converting from sound to text to graphics 185 VII Movies 187 10 Encoding, Manipulation and Creating Movies 189 10.1 Manipulating movies 189 10.2 Compositing to create new movies 191 10.3 Animation: Creating movies from scratch 192 11 Storing Media 195 VIII 197 Isn’t there a better way? 12 How fast can we get? 199 12.1 Complexity 199 12.2 Speed limits 199 12.3 Native code versus interpreted code 200 IX Programming and Design 201 13 Functional Decomposition 203 13.1 Using map 203 14 Recursion 205 15 Objects 207 X 209 Other Languages 16 Java 211 16.1 Java examples 211 vi CONTENTS List of Figures 1.1 1.2 Eight wires with a pattern of voltages is a byte, which gets interpreted as a pattern of eight 0’s and 1’s, which gets interpreted as a decimal number Alan Perlis 12 17 2.1 2.2 2.3 2.4 2.5 2.6 JES running on MacOS X (with annotations) JES running on Windows The File Picker File picker with media types identified Picking, making, and showing a picture Defining and executing pickAndShow 24 25 28 29 30 34 3.1 Raindrops causing ripples in the surface of the water, just as sound causes ripples in the air One cycle of the simplest sound, a sine wave The distance between peaks in ripples in a pond are not constant—some gaps are longer, some shorter The note A above middle C is 440 Hz Some synthesizers using triangular (or sawtooth) or square waves Sound editor main tool Viewing the sound signal as it comes in Viewing the sound in a spectrum view Viewing a sound in spectrum view with multiple “spikes” Viewing the sound signal in a sonogram view Area under a curve estimated with rectangles A depiction of the first five elements in a real sound array A sound recording graphed in the MediaTools The sound editor open menu 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 vii 46 46 48 49 50 50 51 52 53 54 55 57 57 58 LIST OF FIGURES viii 3.15 3.16 3.17 3.18 MediaTools open file dialog A sound opened in the editor Exploring the sound in the editor Comparing the graphs of the original sound (top) and the louder one (bottom) 3.19 Comparing specific samples in the original sound (top) and the louder one (bottom) 3.20 Comparing the original sound (top) to the spliced sound (bottom) 4.1 4.2 4.3 4.4 4.5 4.6 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.11 5.12 5.13 5.14 5.15 The top and middle waves are added together to create the bottom wave The raw 440 Hz signal on top, then the 440+880+1320 Hz signal on the bottom FFT of the 440 Hz sound FFT of the combined sound The 440 Hz square wave (top) and additive combination of square waves (bottom) FFT’s of the 440 Hz square wave (top) and additive combination of square waves (bottom) An example matrix Cursor and icon at regular magnification on top, and close-up views of the cursor (left) and the line below the cursor (right) Merging red, green, and blue to make new colors The Macintosh OS X RGB color picker Picking a color using RGB sliders from JES RGB triplets in a matrix representation Directly modifying the pixel colors via commands: Note the small yellow line on the left Using the MediaTools image exploration tools The original picture (left) and red-reduced version (right) Overly blue (left) and red increased by 20% (right) Original (left) and blue erased (right) Lightening and darkening of original picture Negative of the image Color picture converted to greyscale Once we pick a mirrorpoint, we can just walk x halfway and subtract/add to mirrorpoint 58 59 59 69 70 86 109 110 110 110 111 111 116 117 118 118 119 120 125 126 128 129 130 131 132 134 136 LIST OF FIGURES ix 5.16 Original picture (left) and mirrored along the vertical axis (right) 137 5.17 Flowers in the mediasources folder 141 5.18 Collage of flowers 142 5.19 Increasing reds in the browns 147 5.20 Increasing reds in the browns, within a certain range 148 5.21 A picture of a child (Katie), and her background without her 149 5.22 A new background, the moon 149 5.23 Katie on the moon 150 5.24 Mark in front of a blue sheet 150 5.25 Mark on the moon 151 5.26 Mark in the jungle 152 5.27 Color: RGB triplets in a matrix representation 159 5.28 Color: The original picture (left) and red-reduced version (right)160 5.29 Color: Overly blue (left) and red increased by 20% (right) 160 5.30 Color: Original (left) and blue erased (right) 161 5.31 Color: Lightening and darkening of original picture 161 5.32 Color: Negative of the image 162 5.33 Color: Color picture converted to greyscale 162 5.34 Color: Increasing reds in the browns 163 5.35 Color: Increasing reds in the browns, within a certain range 164 6.1 6.2 A very small, drawn picture 166 A very small, drawn picture 167 7.1 Seeing the variables using showVars() 173 10.1 Movie tools in MediaTools 189 10.2 Dark starting frame number 190 10.3 Somewhat lighter starting frame number 190 ... twice as much computing power every 18 months That means, in a year-and -a- half, computers get as fast over again as has taken them since World War II This Law has continued to hold true for decades... can teach a computer that cells contain mitochondria and DNA, and that DNA has four kinds of nucleotides, and that factories have these kinds of presses and these kinds of stamps Creating layers.. .Introduction to Media Computation: A Multimedia Cookbook in Python Mark Guzdial December 16, 2002 ii Contents I Introduction Introduction to Media Computation 1.1 What is computer science about?