MEAP Edition Manning Early Access Program Math for Programmers 3D graphics, machine learning, and simulations with Python Version 10 Copyright 2020 Manning Publications ©Manning Publications Co We welcome reader comments about anything in the manuscript - other than typos and other simple mistakes These will be cleaned up during production of the book by copyeditors and proofreaders https://livebook.manning.com/#!/book/math-for-programmers/discussion For more information on this and other Manning titles go to www.manning.com welcome Thank you for purchasing the MEAP of Math for Programmers I’ve been a math enthusiast my whole life, and only accidentally stumbled into software engineering as a career I first taught myself how to code on a TI-84 graphing calculator, writing programs to my high school math homework for me Ever since then I’ve been excited by how complementary the disciplines of math and programming can be I look forward to sharing what I’ve learned with you! Now more than ever, knowing some math can accelerate your career as a developer As evidence, look at the recent prevalence of the job title “Data Scientist,” and what people with this title get paid I joke with my coworkers about what a “Data Scientist” really is, but the best answer is probably someone who knows statistics, linear algebra, and calculus, and how to turn them into code Beyond data analysis, these fields of math are useful in graphics, game design, simulation, optimization, and many other software development domains In this book, we’ll start by exploring vectors, which are the mathematical tool for representing multidimensional data Computer graphics in 2D and 3D are built with vectors, and you’ll learn how to render your own 3D animations using matrix transformations Part culminates by showing you how these geometric lessons extend to higher dimensions within the framework of linear algebra Part focuses on calculus, which is the study of continuous change You’ll learn that many of the laws of physics can be expressed in terms of calculus equations called differential equations By solving these in Python, you can create realistic simulations of the physical world With a working knowledge of calculus and linear algebra, you’ll be ready to learn some of the math behind machine learning in Part Machine learning algorithms are often used to draw conclusions about vector data, and the “learning” is often accomplished using an operation from calculus called the gradient With a title as broad as Math for Programmers, there’s more content than I’ll be able to cover I look forward to hearing your feedback on what I have included as well as what I haven’t Please post your questions and comments in the forum, and I will take them seriously to make this book as useful as possible Happy reading! —Paul Orland brief contents Learning Math in Code Part I: Vectors and Graphics Drawing with 2D Vectors Ascending to the 3D World Transforming Vectors and Graphics Computing Transformations with Matrices Generalizing to Higher Dimensions Solving Systems of Linear Equations Part 2: Calculus and Physical Simulation Understanding rates of change Simulating Moving Objects 10 Working with symbolic expressions 11 Simulating Force Fields 12 Optimizing a physical system 13 Analyzing sound waves with Fourier series Part 3: Machine Learning Applications 14 Fitting functions to data 15 Classifying data with logistic regression 16 Training neural networks Appendixes: Appendix A: Loading and Rendering 3D Models with OpenGL and PyGame Appendix B: Getting set up with Python Learning Math in Code This chapter covers · · · · Making money by implementing mathematical ideas in code Avoiding common pitfalls in math learning Thinking like a programmer to understand math Using Python as a powerful and extensible calculator Math is like baseball or poetry or fine wine Some people are so fascinated by math that they devote their whole lives to it, while others feel like they just don’t “get it.” You’ve probably already been forced into one camp or another by twelve years of compulsory math education in school What if we learned about fine wine in school like we learn math? I don’t think I’d like wine at all if I got lectured on grape varietals and fermentation techniques for an hour a day Maybe in such a world, I’d need to consume three or four glasses for homework, as assigned by the teacher Sometimes this would be a delicious and educational experience, but sometimes I wouldn’t feel like getting loaded on a school night My experience in math class went something like this, and it turned me off of the subject for a while Like wine, mathematics is an acquired taste, and a daily grind of lectures and assignments is no way to refine one’s palate If you miss this, it’s easy to think you’re either “cut out” for math or you aren’t If you already believe in yourself and you’re excited to start learning, that’s great! Otherwise, this chapter is designed for you Feeling intimidated by math is so common, it has a name: “math anxiety.” I hope to dispel any anxiety you might have, and show you that math can be a stimulating experience rather than a frightening one All you need are the right tools and the right mindset The main tool for learning in this book is Python programming I’m guessing when you learned math in high school, you saw it written on the blackboard and not written in computer code This is a shame, because a high-level programming language is a far more powerful than a blackboard, and far more versatile than whatever overpriced graphing calculator you may have used An advantage of meeting math in code is that the ideas have to be precise enough for a computer to understand, there’s never any hand-waving about what new symbols mean As with learning any new subject, the best way to set yourself up for success is to want to learn it There are plenty of good reasons You could be intrigued by the beauty of mathematical concepts or enjoy the “brain-teaser” feel of math problems Maybe there’s an app or game you’ve been dreaming of building that needs some math to make it work For now, I’ll try to motivate you with an even lower common denominator: solving mathematical problems with software can make you filthy rich 1.1 Solving lucrative problems with math and software A classic criticism you hear in high school math class is “when am I ever going to use this stuff in real life?” Our teachers told us that math would help us succeed professionally and make money I think they were right about this, even though their examples were off For instance, I don’t calculate my compounding bank interest by hand (and neither does my bank) Maybe if I became a construction site surveyor, as my trigonometry teacher suggested, I’d be using sines and cosines every day to earn my paycheck It turns out the “real world” applications from high school textbooks aren’t that useful Still, there are real applications of math out there, and some of them are mind-bogglingly lucrative Many of them are solved by translating the right mathematical idea into usable software I’ll share some of my favorite examples 1.1.1 Predicting financial market movements We’ve all heard legends of stock traders making millions of dollars by buying and selling the right stocks at the right time Based on the movies I’ve seen, I always pictured a trader as a middle-aged man in a suit yelling at his broker over a cell phone while driving around in a sports car Maybe this stereotype was spot-on at one point, but the situation is different today Holed up in back-offices of skyscrapers all over Manhattan are thousands of people called quants Quants, otherwise known as quantitative analysts, design mathematical algorithms that automatically trade stocks and earn a profit They don’t wear suits and they don’t spend any time yelling on their cell phones, but I’m sure many of them still own very nice sports cars So how does a quant write a program that automatically makes money? The best answers to this question are closely-guarded trade secrets, but you can be sure they involve a lot of math We can look at a toy example to get a sense of how an automated trading strategy might work Stocks represents an ownership stakes in companies When the market perceives a company is doing well, the price goes up: buying the stock becomes more costly and selling it becomes more rewarding Stock prices change erratically and in real time A graph of a stock price over a day of trading might look something like this Figure 1.1 Typical graph of a stock price over time If you bought a thousand shares of this stock for $24 around minute 100 and sold them for $38 at minute 400, you would make off with $14,000 for the day Not bad! The challenge is that you’d have to know in advance that the stock was going up, and that minutes 100 and 400 were the best times to buy and sell, respectively It may not be possible to predict the exact lowest highest price points, but maybe you can find relatively good times to buy and sell throughout the day Let’s look at a way to this mathematically First we could measure whether the stock is going up or down by finding a line of “best fit”, that approximately follows the direction the price is moving This process is called linear regression, and we’ll cover it in part of the book Based on the variability of data, we can calculate two more lines above and below the “best fit” line that show the region in which the price is wobbling up and down Overlaid on the price graph, we see they follow the trend nicely Figure 1.2 Using linear regression to identify the trend in a changing stock price With a mathematical understanding of the price movement, we could write software to automatically buy when the price is going through a low fluctuation and automatically sell when the price goes back up Specifically, our program could connect to the stock exchange over the network and buy 100 shares whenever the price crosses the bottom line and sell 100 shares whenever the price crosses the top line One profitable trade is shown below: entering at around $27.80 and selling at around $32.60 makes you $480 in an hour Figure B.21 If you redefine a symbol like f below and then re-run a box above, it will use the new definition This can be confusing, but at least Jupyter re-numbers the input boxes as you run them For reproducibility, I suggest you define variables and functions above their first usage You can confirm your code runs correctly from top to bottom by clicking the menu item “Kernel > Restart & Run All.” This will wipe out all of your existing computations, but if you’ve stayed organized you should get the same results back Figure B.22 Use “Restart & Run All” to clear the outputs and run all of your inputs from top to bottom Your notebook is automatically saved as you go, but when you’re done coding you can name it by clicking “Untitled” at the top and entering a new name Figure B.23 Giving your notebook a name Then you can click the “Jupyter” logo once again to return to the main menu, and you should see your new notebook saved as a file with the “.ipynb” extension If you want to return to your notebook, you can click its name to open it Figure B.24 Your new notebook appears To make sure all your files are saved, you should exit Jupyter by clicking Quit, rather than just closing the browser tab or stopping the interactive process For more info on Jupyter notebooks, you can consult the extensive documentation on https://jupyter.org/ At this point, however, you know enough to download and play with the source code for the book, which are mostly in Jupyter form Table of Contents Copyright 2020 Manning Publications welcome brief contents Learning Math in Code 1.1 Solving lucrative problems with math and software 1.1.1 Predicting financial market movements 1.1.2 Finding a good deal 1.1.3 Building 3D graphics and animations 1.1.4 Modeling the physical world 1.2 How not to learn math 1.2.1 Jane wants to learn some math 1.2.2 Slogging through math textbooks 1.3 Using your well-trained left brain 1.3.1 Using a formal language 1.3.2 Build your own calculator 1.3.3 Building abstractions with functions 1.4 Summary Drawing with 2D Vectors 2.1 Drawing with 2D Vectors 2.1.1 Representing 2D vectors 2.1.2 2D Drawing in Python 2.1.3 Exercises 2.2 Plane vector arithmetic 2.2.1 Vector components and lengths 2.2.2 Multiplying Vectors by Numbers 2.2.3 Subtraction, displacement, and distance 2.2.4 Exercises 2.3 Angles and Trigonometry in the Plane 2.3.1 From angles to components 2.3.2 Radians and trigonometry in Python 2.3.3 From components back to angles 2.3.4 Exercises 2.4 Transforming collections of vectors 2.4.1 Combining vector transformations 2.4.2 Exercises 2.5 Drawing with Matplotlib 2.6 Summary Ascending to the 3D World 3.1 Picturing vectors in three-dimensional space 3.1.1 Representing 3D vectors with coordinates 3.1.2 3D Drawing in Python 3.1.3 Exercises 3.2 Vector arithmetic in 3D 3.2.1 Adding 3D vectors 3.2.2 Scalar Multiplication in 3D 3.2.3 Subtracting 3D vectors 3.2.4 Computing lengths and distances 3.2.5 Computing angles and directions 3.2.6 Exercises 3.3 The dot product: measuring alignment of vectors 3.3.1 Picturing the dot product 3.3.2 Computing the dot product 3.3.3 Dot products by example 3.3.4 Measuring angles with the dot product 3.3.5 Exercises 3.4 The cross product: measuring oriented area 3.4.1 Orienting ourselves in 3D 3.4.2 Finding the direction of the cross product 3.4.3 Finding the length of the cross product 3.4.4 Computing the cross product of 3D vectors 3.4.5 Exercises 3.5 Rendering a 3D object in 2D 3.5.1 Defining a 3D object with vectors 3.5.2 Projecting to 2D 3.5.3 Orienting faces and shading 3.5.4 Exercises 3.6 Summary Transforming Vectors and Graphics 4.1 Transforming 3D objects 4.1.1 Drawing a transformed object 4.1.2 Composing vector transformations 4.1.3 Rotating an object about an axis 4.1.4 Inventing your own geometric transformations 4.1.5 Exercises 4.2 Linear transformations 4.2.1 Preserving vector arithmetic 4.2.2 Picturing linear transformations 4.2.3 Why linear transformations? 4.2.4 Computing linear transformations 4.2.5 Exercises 4.3 Summary Computing Transformations with Matrices 5.1 Representing linear transformations with matrices 5.1.1 Writing vectors and linear transformations as matrices 5.1.2 Multiplying a matrix with a vector 5.1.3 Composing linear transformations by matrix multiplication 5.1.4 Implementing matrix multiplication 5.1.5 3D Animation with matrix transformations 5.1.6 Exercises 5.2 Interpreting matrices of different shapes 5.2.1 Column vectors as matrices 5.2.2 What pairs of matrices can be multiplied? 5.2.3 Viewing square and non-square matrices as vector functions 5.2.4 Projection as a linear map from 3D to 2D 5.2.5 Composing linear maps 5.2.6 Exercises 5.3 Translating vectors with matrices 5.3.1 Making plane translations linear 5.3.2 Finding a 3D matrix for a 2D translation 5.3.3 Combining translation with other linear transformations 5.3.4 Translating 3D objects in a 4D world 5.3.5 Exercises 5.4 Summary Generalizing to Higher Dimensions 6.1 Generalizing our definition of vectors 6.1.1 Creating a class for 2D coordinate vectors 6.1.2 Improving the Vec2 class 6.1.3 Repeating the process with 3D vectors 6.1.4 Building a Vector base class 6.1.5 Defining vector spaces 6.1.6 Unit testing vector space classes 6.1.7 Exercises 6.2 Exploring different vector spaces 6.2.1 Enumerating all coordinate vector spaces 6.2.2 Identifying vector spaces in the wild 6.2.3 Treating functions as vectors 6.2.4 Treating matrices as vectors 6.2.5 Manipulating images with vector operations 6.2.6 Exercises 6.3 Looking for smaller vector spaces 6.3.1 Identifying subspaces 6.3.2 Starting with a single vector 6.3.3 Spanning a bigger space 6.3.4 Defining the word “dimension” 6.3.5 Finding subspaces of the vector space of functions 6.3.6 Subspaces of images 6.3.7 Exercises 6.4 Summary Solving Systems of Linear Equations 7.1 Designing an arcade game 7.1.1 Modeling the game 7.1.2 Rendering the game 7.1.3 Shooting the laser 7.1.4 Exercises 7.2 Finding intersection points of lines 7.2.1 Choosing the right formula for a line 7.2.2 Finding the standard form equation for a line 7.2.3 Linear equations in matrix notation 7.2.4 Solving linear equations with numpy 7.2.5 Deciding whether the laser hits an asteroid 7.2.6 Identifying unsolvable systems 7.2.7 Exercises 7.3 Generalizing linear equations to higher dimensions 7.3.1 Representing planes in 3D 7.3.2 Solving linear equations in 3D 7.3.3 Studying hyperplanes algebraically 7.3.4 Counting dimensions, equations, and solutions 7.3.5 Exercises 7.4 Changing basis by solving linear equations 7.4.1 Solving a 3D example 7.4.2 Exercises 7.5 Summary Understanding Rates of Change 8.1 Calculating average flow rates from volumes 8.1.1 Implementing an average_flow_rate function 8.1.2 Picturing the average flow rate with a secant line 8.1.3 Negative rates of change 8.1.4 Exercises 8.2 Plotting the average flow rate over time 8.2.1 Finding the average flow rate in different time intervals 8.2.2 Plotting the interval flow rates alongside the flow rate function 8.2.3 Exercises 8.3 Approximating instantaneous flow rates 8.3.1 Finding the slope of very small secant lines 8.3.2 Building the instantaneous flow rate function 8.3.3 Currying and plotting the instantaneous flow rate function 8.3.4 Exercises 8.4 Approximating the change in volume 8.4.1 Finding the change in volume on a short time interval 8.4.2 Breaking up time into small intervals 8.4.3 Picturing the volume change on the flow rate graph 8.4.4 Exercises 8.5 Plotting the volume over time 8.5.1 Finding the volume over time 8.5.2 Picturing Riemann sums for the volume function 8.5.3 Improving the approximation 8.5.4 Definite and indefinite integrals 8.6 Summary Simulating Moving Objects 9.1 Simulating constant-speed motion 9.1.1 Intuiting speed 9.1.2 Thinking of velocity as a vector 9.1.3 Animating the asteroids 9.1.4 Exercises 9.2 Simulating acceleration 9.2.1 Picturing acceleration in various directions 9.2.2 Quantifying Acceleration 9.2.3 Accelerating the spaceship 9.2.4 Exercises 9.3 Digging deeper into Euler’s method 9.3.1 Stepping through Euler’s method 9.3.2 Implementing the algorithm in Python 9.3.3 Picturing the approximation 9.3.4 Applying Euler’s method to other problems 9.3.5 Exercises 9.4 Calculating exact trajectories 9.4.1 Writing position, velocity, and acceleration as functions of time 9.4.2 Using the terminology of integration 9.4.3 Calculating integrals 9.4.4 Exercises 9.5 Summary 10 Working with Symbolic Expressions 10.1 Modeling algebraic expressions 10.1.1 Breaking an expression into pieces 10.1.2 Building an expression tree 10.1.3 Translating the expression tree to Python 10.1.4 Exercises 10.2 Putting a symbolic expression to work 10.2.1 Finding all the variables in an expression 10.2.2 Evaluating an expression 10.2.3 Expanding an expression 10.2.4 Exercises 10.3 Finding the derivative of a function 10.3.1 Derivatives of powers 10.3.2 Derivatives of transformed functions 10.3.3 Derivatives of some special functions 10.3.4 Derivatives of products and compositions 10.3.5 Exercises 10.4 Taking derivatives automatically 10.4.1 Implementing a derivative method for expressions 10.4.2 Implementing the product rule and chain rule 10.4.3 Implementing the power rule 10.4.4 Exercises 10.5 Integrating functions symbolically 10.5.1 Integrals as antiderivatives 10.5.2 Introducing the SymPy library 10.5.3 Exercises 10.6 Summary 11 Simulating Force Fields 11.1 Modeling gravitational fields 11.1.1 Defining a vector field 11.1.2 Defining a simple force field 11.2 Adding gravity to the asteroid game 11.2.1 Making game objects feel gravity 11.2.2 Exercises 11.3 Introducing potential energy 11.3.1 Defining a potential energy scalar field 11.3.2 Plotting a scalar field as a heatmap 11.3.3 Plotting a scalar field as a contour map 11.4 Connecting energy and forces with the gradient 11.4.1 Measuring steepness with cross sections 11.4.2 Calculating partial derivatives 11.4.3 Finding the steepness of a graph with the gradient 11.4.4 Calculating force fields from potential energy with the gradient 11.4.5 Exercises 11.5 Summary 12 Optimizing a Physical System 12.1 Testing a projectile simulation 12.1.1 Building a simulation with Euler’s method 12.1.2 Measuring properties of the trajectory 12.1.3 Exploring different launch angles 12.1.4 Exercises 12.2 Calculating the optimal range 12.2.1 Finding the projectile range as a function of the launch angle 12.2.2 Solving for the maximum range 12.2.3 Identifying maxima and minima 12.2.4 Exercises 12.3 Enhancing our simulation 12.3.1 Adding another dimension 12.3.2 Modeling terrain around the cannon 12.3.3 Solving for the range of the projectile in 3D 12.3.4 Exercises 12.4 Optimizing range using gradient ascent 12.4.1 Plotting range versus launch parameters 12.4.2 The gradient of the range function 12.4.3 Finding the uphill direction with the gradient 12.4.4 Implementing gradient ascent 12.4.5 Exercises 12.5 Summary 13 Analyzing sound waves with Fourier series Combining sound waves and decomposing them 13.1 Playing sound waves in Python 13.1.1 Producing our first sound 13.1.2 Playing a musical note 13.1.3 Exercises 13.2 Turning a sinusoidal wave into a sound 13.2.1 Making audio from sinusoidal functions 13.2.2 Changing the frequency of a sinusoid 13.2.3 Sampling and playing the sound wave 13.2.4 Exercises 13.3 Combining sound waves to make new ones 13.3.1 Adding sampled sound waves to build a chord 13.3.2 Picturing the sum of two sound waves 13.3.3 Building a linear combination of sinusoids 13.3.4 Building a familiar function with sinusoids 13.3.5 Exercises 13.4 Decomposing a sound wave into its Fourier Series 13.4.1 Finding vector components with an inner product 13.4.2 Defining an inner product for periodic functions 13.4.3 Writing a function to find Fourier coefficients 13.4.4 Finding the Fourier coefficients for the square wave 13.4.5 Fourier coefficients for other waveforms 13.4.6 Exercises 13.5 Summary 14 Fitting functions to data 14.1 Measuring the quality of fit for a function 14.1.1 Measuring distance from a function 14.1.2 Summing the squares of the errors 14.1.3 Calculating cost for car price functions 14.1.4 Exercises 14.2 Exploring spaces of functions 14.2.1 Picturing cost for lines through the origin 14.2.2 The space of all linear functions 14.2.3 Exercises 14.3 Finding the line of best fit using gradient descent 14.3.1 Rescaling the data 14.3.2 Finding and plotting the line of best fit 14.3.3 Exercises 14.4 Fitting a nonlinear function 14.4.1 Understanding the behavior of exponential functions 14.4.2 Finding the exponential function of best fit 14.4.3 Exercises 14.5 Summary 15 Classifying data with logistic regression 15.1 Testing a classification function on real data 15.1.1 Loading the car data 15.1.2 Testing the classification function 15.1.3 Exercises 15.2 Picturing a decision boundary 15.2.1 Picturing the space of cars 15.2.2 Drawing a better decision boundary 15.2.3 Implementing the classification function 15.2.4 Exercises 15.3 Framing classification as a regression problem 15.3.1 Scaling the raw car data 15.3.2 Measuring BMWness of a car 15.3.3 Introducing the sigmoid function 15.3.4 Composing the sigmoid function with other functions 15.3.5 Exercises 15.4 Exploring possible logistic functions 15.4.1 Parameterizing logistic functions 15.4.2 Measuring the quality of fit for a logistic function 15.4.3 Testing different logistic functions 15.4.4 Exercises 15.5 Finding the best logistic function 15.5.1 Gradient descent in three dimensions 15.5.2 Using gradient descent to find the best fit 15.5.3 Testing an understanding the best logistic classifier 15.5.4 Exercises 15.6 Summary 16 Training neural networks 16.1 Classifying data with neural networks 16.2 Classifying images of handwritten digits 16.2.1 Building the 64-dimensional image vectors 16.2.2 Building a random digit classifier 16.2.3 Measuring performance of the digit classifier 16.2.4 Exercises 16.3 Designing a neural network 16.3.1 Organizing neurons and connections 16.3.2 Data flow through a neural network 16.3.3 Calculating activations 16.3.4 Calculating activations in matrix notation 16.3.5 Exercises 16.4 Building a neural network in Python 16.4.1 Implementing an MLP class in Python 16.4.2 Evaluating the MLP 16.4.3 Testing the classification performance of an MLP 16.4.4 Exercises 16.5 Training a neural network using gradient descent 16.5.1 Framing training as a minimization problem 16.5.2 Calculating gradients with backpropagation 16.5.3 Automatic training with scikit-learn 16.5.4 Exercises 16.6 Calculating gradients with backpropagation 16.6.1 Finding the cost in terms of the last layer weights 16.6.2 Calculating the partial derivatives for the last layer weights using the chain rule 16.6.3 Exercises 16.7 Summary Appendix A Loading and Rendering 3D Models with OpenGL and PyGame A.1 Recreating the octahedron from Chapter A.2 Changing our perspective A.3 Loading and rendering the Utah teapot A.4 Exercises Appendix B Getting set up with Python B.1 Checking for an existing Python installation B.2 Downloading and installing Anaconda B.3 Using Python in interactive mode B.4 Creating and running a Python script file B.5 Using Jupyter notebooks .. .MEAP Edition Manning Early Access Program Math for Programmers 3D graphics, machine learning, and simulations with Python Version 10 Copyright 2020 Manning... For more information on this and other Manning titles go to www.manning.com welcome Thank you for purchasing the MEAP of Math for Programmers I’ve been a math enthusiast my whole life, and only... the mathematical tool for representing multidimensional data Computer graphics in 2D and 3D are built with vectors, and you’ll learn how to render your own 3D animations using matrix transformations