©Manning Publications Co To comment go to liveBook MEAP Edition Manning Early Access Program Math for Programmers 3D graphics, machine learning, and simulations with Python Version 10 Copyright 2020 Manning Publications For more information on this and other Manning titles go to manning.com ©Manning Publications Co To comment go to liveBook ©Manning Publications Co To comment go to liveBook welcome Thank you for purchasing the MEAP of Math for Programmers I’ve been a math enthusiast my whole life, and only accidentally stumbled into software engineering as a career I first taught myself how to code on a TI-84 graphing calculator, writing programs to my high school math homework for me Ever since then I’ve been excited by how complementary the disciplines of math and programming can be I look forward to sharing what I’ve learned with you! Now more than ever, knowing some math can accelerate your career as a developer As evidence, look at the recent prevalence of the job title “Data Scientist,” and what people with this title get paid I joke with my coworkers about what a “Data Scientist” really is, but the best answer is probably someone who knows statistics, linear algebra, and calculus, and how to turn them into code Beyond data analysis, these fields of math are useful in graphics, game design, simulation, optimization, and many other software development domains In this book, we’ll start by exploring vectors, which are the mathematical tool for representing multidimensional data Computer graphics in 2D and 3D are built with vectors, and you’ll learn how to render your own 3D animations using matrix transformations Part culminates by showing you how these geometric lessons extend to higher dimensions within the framework of linear algebra Part focuses on calculus, which is the study of continuous change You’ll learn that many of the laws of physics can be expressed in terms of calculus equations called differential equations By solving these in Python, you can create realistic simulations of the physical world With a working knowledge of calculus and linear algebra, you’ll be ready to learn some of the math behind machine learning in Part Machine learning algorithms are often used to draw conclusions about vector data, and the “learning” is often accomplished using an operation from calculus called the gradient With a title as broad as Math for Programmers, there’s more content than I’ll be able to cover haven’t I look forward to hearing your feedback on what I have included as well as what I Please post your questions and comments in the forum, and I will take them seriously to make this book as useful as possible Happy reading! —Paul Orland ©Manning Publications Co To comment go to liveBook brief contents Learning Math in Code PART 1: VECTORS AND GRAPHICS Drawing with 2D Vectors Ascending to the 3D World Transforming Vectors and Graphics Computing Transformations with Matrices Generalizing to Higher Dimensions Solving Systems of Linear Equations PART 2: CALCULUS AND PHYSICAL SIMULATION Understanding rates of change Simulating Moving Objects 10 Working with symbolic expressions 11 Simulating Force Fields 12 Optimizing a physical system 13 Analyzing sound waves with Fourier series PART 3: MACHINE LEARNING APPLICATIONS 14 Fitting functions to data 15 Classifying data with logistic regression 16 Training neural networks APPENDIXES: A: Loading and Rendering 3D Models with OpenGL and PyGame B: Getting set up with Python ©Manning Publications Co To comment go to liveBook 1 Learning Math in Code This chapter covers • Making money by implementing mathematical ideas in code • Avoiding common pitfalls in math learning • Thinking like a programmer to understand math • Using Python as a powerful and extensible calculator Math is like baseball or poetry or fine wine Some people are so fascinated by math that they devote their whole lives to it, while others feel like they just don’t “get it.” You’ve probably already been forced into one camp or another by twelve years of compulsory math education in school What if we learned about fine wine in school like we learn math? I don’t think I’d like wine at all if I got lectured on grape varietals and fermentation techniques for an hour a day Maybe in such a world, I’d need to consume three or four glasses for homework, as assigned by the teacher Sometimes this would be a delicious and educational experience, but sometimes I wouldn’t feel like getting loaded on a school night My experience in math class went something like this, and it turned me off of the subject for a while Like wine, mathematics is an acquired taste, and a daily grind of lectures and assignments is no way to refine one’s palate If you miss this, it’s easy to think you’re either “cut out” for math or you aren’t If you already believe in yourself and you’re excited to start learning, that’s great! Otherwise, this chapter is designed for you Feeling intimidated by math is so common, it has a name: “math anxiety.” I hope to dispel any anxiety you might have, and show you that math can be a stimulating experience rather than a frightening one All you need are the right tools and the right mindset The main tool for learning in this book is Python programming I’m guessing when you learned math in high school, you saw it written on the blackboard and not written in computer ©Manning Publications Co To comment go to liveBook code This is a shame, because a high-level programming language is a far more powerful than a blackboard, and far more versatile than whatever overpriced graphing calculator you may have used An advantage of meeting math in code is that the ideas have to be precise enough for a computer to understand, there’s never any hand-waving about what new symbols mean As with learning any new subject, the best way to set yourself up for success is to want to learn it There are plenty of good reasons You could be intrigued by the beauty of mathematical concepts or enjoy the “brain-teaser” feel of math problems Maybe there’s an app or game you’ve been dreaming of building that needs some math to make it work For now, I’ll try to motivate you with an even lower common denominator: solving mathematical problems with software can make you filthy rich 1.1 Solving lucrative problems with math and software A classic criticism you hear in high school math class is “when am I ever going to use this stuff in real life?” Our teachers told us that math would help us succeed professionally and make money I think they were right about this, even though their examples were off For instance, I don’t calculate my compounding bank interest by hand (and neither does my bank) Maybe if I became a construction site surveyor, as my trigonometry teacher suggested, I’d be using sines and cosines every day to earn my paycheck It turns out the “real world” applications from high school textbooks aren’t that useful Still, there are real applications of math out there, and some of them are mind-bogglingly lucrative Many of them are solved by translating the right mathematical idea into usable software I’ll share some of my favorite examples 1.1.1 Predicting financial market movements We’ve all heard legends of stock traders making millions of dollars by buying and selling the right stocks at the right time Based on the movies I’ve seen, I always pictured a trader as a middle-aged man in a suit yelling at his broker over a cell phone while driving around in a sports car Maybe this stereotype was spot-on at one point, but the situation is different today Holed up in back-offices of skyscrapers all over Manhattan are thousands of people called quants Quants, otherwise known as quantitative analysts, design mathematical algorithms that automatically trade stocks and earn a profit They don’t wear suits and they don’t spend any time yelling on their cell phones, but I’m sure many of them still own very nice sports cars So how does a quant write a program that automatically makes money? The best answers to this question are closely-guarded trade secrets, but you can be sure they involve a lot of math We can look at a toy example to get a sense of how an automated trading strategy might work Stocks represents an ownership stakes in companies When the market perceives a company is doing well, the price goes up: buying the stock becomes more costly and selling it becomes more rewarding Stock prices change erratically and in real time A graph of a stock price over a day of trading might look something like this ©Manning Publications Co To comment go to liveBook Figure 1.1 Typical graph of a stock price over time If you bought a thousand shares of this stock for $24 around minute 100 and sold them for $38 at minute 400, you would make off with $14,000 for the day Not bad! The challenge is that you’d have to know in advance that the stock was going up, and that minutes 100 and 400 were the best times to buy and sell, respectively It may not be possible to predict the exact lowest highest price points, but maybe you can find relatively good times to buy and sell throughout the day Let’s look at a way to this mathematically First we could measure whether the stock is going up or down by finding a line of “best fit”, that approximately follows the direction the price is moving This process is called linear regression, and we’ll cover it in part of the book Based on the variability of data, we can calculate two more lines above and below the “best fit” line that show the region in which the price is wobbling up and down Overlaid on the price graph, we see they follow the trend nicely ©Manning Publications Co To comment go to liveBook Figure 1.2 Using linear regression to identify the trend in a changing stock price With a mathematical understanding of the price movement, we could write software to automatically buy when the price is going through a low fluctuation and automatically sell when the price goes back up Specifically, our program could connect to the stock exchange over the network and buy 100 shares whenever the price crosses the bottom line and sell 100 shares whenever the price crosses the top line One profitable trade is shown below: entering at around $27.80 and selling at around $32.60 makes you $480 in an hour Figure 1.3 Buying and selling according to a rule to make a profit ©Manning Publications Co To comment go to liveBook I’m sweeping a bunch of the complexities under the rug here, but the core idea works At this very moment, some unknowable number of programs are building and updating models measuring the predicted trend of stocks and other financial instruments If you write such a program, you can enjoy some leisure time while it makes money for you! 1.1.2 Finding a good deal Maybe you don’t have deep enough pockets to consider risky stock trading Math can still help you make and save money in other transactions, like buying a used car New cars are easy to understand commodities; if two dealers are selling the same car you obviously want to choose the cheaper of the two Used cars have more numbers associated to them: an asking price as well as a mileage and a model year You can even use the duration that a particular used car has been on the market to assess its quality the longer the duration, the more suspicious you might be In mathematics, objects you can describe with ordered lists of numbers are called vectors, and there is a whole field called linear algebra dedicated to studying them A used car might correspond to a four-dimensional vector, meaning a four-tuple of numbers: (2015, 41429, 22.27, 16980) These numbers are the model year, mileage, days on the market, and asking price A friend of mine runs a site called CarGraph.com that aggregates data on used cars for sale At the time of writing, it shows 101 Toyota Priuses for sale, and it gives some or all of these four pieces of data for each one The site also lives up to its name and visually presents the data in a graph It’s hard to visualize four-dimensional objects, but if you choose two of the dimensions like price and mileage, you can plot them as points on a scatter plot ©Manning Publications Co To comment go to liveBook Figure 1.4 A graph of price vs mileage for used priuses from CarGraph.com We might be interested in drawing a trend line here too every point on this graph represents someone’s opinion of a fair price, so the trend line would aggregate these opinions together into a more reliable price at any mileage In this case, I decided to fit to an exponential decline curve rather than a line, and I omitted some of the nearly-new cars selling for below retail price Figure 1.5 Fitting an exponential decline curve to price vs mileage data The equation for the curve of best fit is ©Manning Publications Co To comment go to liveBook 639 A.4 Exercises EXERCISES Modify the draw_model function to be able to display the input figure from any rotated perspective Specifically, give the draw_model function a keyword argument glRotatefArgs which provides a tuple of four numbers corresponding to the four arguments of glRotatef With this extra information, add an appropriate call to glRotatef within the body of draw_model to execute the rotation SOLUTION In the source code, see draw_model.py for the solution and draw_teapot_glrotatef.py for example usage EXERCISE If we call glRotatef(1,1,1,1) every frame, how many seconds does it take for the scene to complete a full revolution? SOLUTION The answer depends on the framerate This call to glRotatef rotates the perspective by one degree each frame At 60 frames per second, it would rotate 60 degrees per second, and complete a full rotation of 360 degrees in seconds MINI-PROJECT Implement the load_triangles() function from above, which loads the teapot from the teapot.off and produces a list of triangles in Python Each triangle should be specified by a three 3D vectors Then, pass your result to draw_model() as above and confirm that you see the same result SOLUTION In the source code, you can find load_triangles() implemented in the file teapot.py As a hint, you can turn the quadrilaterals into pairs of triangles by connecting their opposite vertices Figure A.6 Indexing four vertices of a quadrilateral, two triangles are formed by vertices 0, 1, and 0, 2, respectively ©Manning Publications Co To comment go to liveBook 640 MINI-PROJECT Animate the teapot by changing the arguments to gluPerspective and glTranslatef This will help you visualize the effects of each of the parameters SOLUTION In animated_octahedron.py in the source code, an example is given for rotating the octahedron by 360/5 = 72 degrees per second by updating the angle parameter of glRotatef every frame You can try similar modifications yourself, with either the teapot or the octahedron ©Manning Publications Co To comment go to liveBook 641 B Getting set up with Python This appendix covers the basic steps to get Python and related tools installed to be able to run the code examples in this book The main thing to install is Anaconda, which is a popular Python distribution for mathematical programming and data science Specifically, Anaconda comes with an interpreter that runs Python code, as well as a number of the most popular math and data science libraries and a coding interface called Jupyter The steps will be mostly the same on any Linux, Mac, or Windows I’m going to show you the steps on my Mac B.1 Checking for an existing Python installation You probably already have Python installed on your Mac by default, and it’s possible you have it already on Windows or Linux even if you didn’t expect it To check, open terminal (or PowerShell on Windows) and type “python” On a Mac, you should see a Python 2.7 terminal appear You can press ctrl + d to exit the terminal Figure B.1 For the examples in this book I’ve used Python 3, which is becoming the new standard, and specifically we’re going to use the Anaconda distribution As a warning, if you have an existing Python installation, the process might be tricky If any of the steps that follow don’t ©Manning Publications Co To comment go to liveBook 642 work for you, my best advice is to search Google or StackOverflow with any error messages you see If you are a Python expert and don’t want to install or use Anaconda, you should be able to find and install the relevant libraries like Numpy, Matplotlib, and Jupyter using the pip package manager For a beginner, I strongly recommend installing Anaconda as follows B.2 Downloading and installing Anaconda Go to https://www.anaconda.com/distribution/ to download the Anaconda Python distribution Click “download” and, choose the Python version beginning with At the time of writing, this was Python 3.7 Figure B.2 At the time of writing, here’s what I see after clicking “download” on the Anaconda website Choose the Python 3.7 download link Open the downloaded installer and it will walk you through the installation process The installer dialog will look different depending on your operating system, but here’s what it looks like on my mac ©Manning Publications Co To comment go to liveBook 643 Figure B.3 The Anaconda installer as it appears on my Mac I used the default installation location and didn’t add any optional features like the PyCharm IDE Once the installation is complete, you should be able to open a fresh terminal, type “python” and enter a Python session with Anaconda Figure B.4 How the Python interactive session should look once Anaconda is installed Notice Python 3.7.3 and the “Anaconda, Inc.” labels that appear If you don’t see a Python version starting with “3” and the word “Anaconda,” it probably means you’re stuck on the previous Python installation on your system You will need to edit your PATH environment variables so that your terminal knows which Python you want when ©Manning Publications Co To comment go to liveBook 644 you type “python” Hopefully you won’t run into this problem, but if so you can search online for a fix B.3 Using Python in interactive mode Three angle brackets “>>>” prompt you to enter a line of Python code When you type 2+2 and press enter, you should see the Python interpreter’s evaluation of this statement, which is Figure B.5 Entering a line of Python in the interactive session This interactive mode is also referred to as a “REPL,” standing for read-evaluate-print loop This Python session reads in a line of typed code, evaluates it, and prints the result, and this process is repeated as many times as you want in a loop Pressing ctrl + d will signify that you’re done entering code, and get you back to your terminal session Python interactive is usually able to identify if you are entering a multi-line statement For instance, “def f(x):” is the first line you’d enter when defining a new Python function called f The Python interactive session shows you “ ” to indicate that it expects more input Figure B.6 The Python interpreter knows you’re not done with your multi-line statement You can then indent, implement the function, and then you’ll need to hit enter twice to let Python know that you’ve finished the multi-line code input Figure B.7 Once you’ve completed your multi-line code, you need to hit enter twice to submit it The function f is now defined in your interactive session, and on the next line you can give it an input to evaluate it Figure B.8 Evaluating the function defined above ©Manning Publications Co To comment go to liveBook 645 Beware that any code you write in an interactive session will disappear when you exit the session For that reason, if you’re writing a lot of code you’ll be better off putting in a script file or in a Jupyter notebook I’ll cover both of these methods next B.4 Creating and running a Python script file You can create Python files with basically any text editor you like As usual, it’s better to use a text editor designed for programming rather than a rich text editor like Microsoft Word, which could insert invisible or unwanted characters for formatting My preference is Visual Studio Code, and other popular choices are Atom, which is cross platform, and Notepad++ for windows At your own risk, you can use a terminal-based text editor like Emacs or Vim All of these tools are free and readily downloadable To create a Python script, simply create a new text file in your editor with a “.py” file extension Here I’ve created a file called “first.py” in my “~/Documents” directory You can see that Visual Studio Code comes with syntax highlighting for Python Keywords, functions, and literal values are colored to make it easy to read the code Many editors (including Visual Studio Code) have optional extensions you can install to give you more helpful tools, like checking for simple errors as you type Here are a few lines of Python typed into first.py Since this is a math book, here’s an example that’s more mathy than “hello world.” When run, it will print the squares of all of the digits from to Figure B.9 Some example Python code in a file Back in the terminal, go to the directory where your Python file lives On my mac, I go to “~/Documents” by typing “cd ~/Documents” You can type “ls first.py” to confirm that you’re in the same directory as your Python script Figure B.10 The “ls” command shows you that first.py is in the directory ©Manning Publications Co To comment go to liveBook 646 To execute the script file, type “python first.py” into the terminal This invokes the Python interpreter and tells it to run the “first.py” file It does what we hoped, and prints out some numbers: Figure B.11 When you’re solving more complicated problems, you may want to break your code up into separate files Next I’ll show you how to put the function f(x) in a different Python file that can be used by first.py Let’s call this new file “function.py”, saving it in the same directory as first.py, and cut and paste the code for f(x) into it Figure B.12 Putting the code to define f(x) in its own Python file To let Python know that we’re going to be combining multiple files in this directory, you’ll have to add an empty text file called “ init .py” in the directory That’s two underscores before and after the word “init” A quick way to create this empty file on a Mac or Linux machine is to type “touch init .py” To use the function f(x) from function.py in the script first.py, we need to let the Python interpreter know to retrieve it To that, we can write “from function import f” as the first line in first.py ©Manning Publications Co To comment go to liveBook 647 Figure B.13 Re-writing the file first.py to include the function When you run “python first.py” again, you should get the same result This time, Python is getting the function f from function.py in the process An alternative to doing all of your work in text files and running them from the command line is using Jupyter notebooks, which I’ll cover next In this book, I most of my examples in Jupyter notebooks, but write any reusable code in separate Python files and import it, as above B.5 Using Jupyter notebooks Jupyter notebooks are graphical interfaces for coding in Python (and other languages as well) As in a Python interactive session, you type lines of code into Jupyter and it prints the result The difference is that Jupyter is a prettier interface than the terminal, and you can save your sessions to resume or re-run later Jupyter notebooks should be automatically installed with Anaconda If you’re using a different Python distribution you can install Jupyter with pip as well See https://jupyter.org/install for documentation if you want to a custom installation To open the Jupyter notebook interface, type “jupyter notebook” or “python -m notebook” in a directory you want to work in You should see a lot of text stream into the terminal, and your default web browser should open showing you the Jupyter notebook interface Here’s what I see in my terminal after typing “python -m notebook” Again, your mileage may vary depending on the Anaconda version you have ©Manning Publications Co To comment go to liveBook 648 Figure B.14 What the terminal looks like when you open a Jupyter notebook Your default web browser should open, showing the Jupyter interface Here’s what I see when Jupyter opens in google chrome Figure B.15 When you start Jupyter, a browser tab should automatically open looking like this What’s going on here is that the terminal is running Python behind the scenes and also serving a local website at the address “localhost:8888” From here on, you only have to think about what’s going on in the browser The code you’ll write in the browser will be automatically sent to the Python process in the terminal via web requests This Python process in the background is called a “kernel” in Jupyter terminology At the first screen that opens in the browser, you can see all of the files in the directory you’re working in For example, I opened the notebook in my “~/Documents” folder, so I can see the Python files we wrote in the last section If you click into one of the files, you’ll see you can view and edit it directly in the web browser Here’s what I see when I click “first.py” ©Manning Publications Co To comment go to liveBook 649 Figure B.16 Jupyter has a basic text editor for Python files This isn’t a notebook yet; a notebook is a different kind of file than an ordinary Python file To create a notebook, return to the main view by clicking the “Jupyter” logo in the top left corner Then go to the “New” dropdown menu on the right, and click “Python 3” Figure B.17 The menu option to create a new Python notebook Once you’ve clicked “Python 3”, you’ll be taken to your new notebook It should look like this, with one blank input line ready to accept some Python code ©Manning Publications Co To comment go to liveBook 650 Figure B.18 A new, empty Jupyter notebook ready for coding You can type a Python expression into the textbox, and then press shift + enter to evaluate it Here I type 2+2 and then press shift + enter to see Figure B.19 Evaluating 2+2 in a Jupyter notebook As you can see, it works just like an interactive session except it looks nicer Each input is shown in a box and the corresponding output is shown below it If you just press “enter” rather than shift + enter, you can add a new line inside your input box Variables and functions defined in boxes above can be used by boxes below Here’s what our original example could look like in a Jupyter notebook ©Manning Publications Co To comment go to liveBook 651 Figure B.20 Writing and evaluating several snippets of Python code in a Jupyter notebook Strictly speaking, each box doesn’t depend on boxes above but on the boxes you previously evaluated If I redefined the function f(x) in the next input box, and then re-ran the previous box, I’d overwrite the previous output ©Manning Publications Co To comment go to liveBook 652 Figure B.21 If you redefine a symbol like f below and then re-run a box above, it will use the new definition This can be confusing, but at least Jupyter re-numbers the input boxes as you run them For reproducibility, I suggest you define variables and functions above their first usage You can confirm your code runs correctly from top to bottom by clicking the menu item “Kernel > Restart & Run All.” This will wipe out all of your existing computations, but if you’ve stayed organized you should get the same results back Figure B.22 Use “Restart & Run All” to clear the outputs and run all of your inputs from top to bottom ©Manning Publications Co To comment go to liveBook 653 Your notebook is automatically saved as you go, but when you’re done coding you can name it by clicking “Untitled” at the top and entering a new name Figure B.23 Giving your notebook a name Then you can click the “Jupyter” logo once again to return to the main menu, and you should see your new notebook saved as a file with the “.ipynb” extension If you want to return to your notebook, you can click its name to open it Figure B.24 Your new notebook appears To make sure all your files are saved, you should exit Jupyter by clicking Quit, rather than just closing the browser tab or stopping the interactive process For more info on Jupyter notebooks, you can consult the extensive documentation on https://jupyter.org/ At this point, however, you know enough to download and play with the source code for the book, which are mostly in Jupyter form ©Manning Publications Co To comment go to liveBook .. .MEAP Edition Manning Early Access Program Math for Programmers 3D graphics, machine learning, and simulations with Python Version 10 Copyright 2020 Manning Publications For more information... ©Manning Publications Co To comment go to liveBook welcome Thank you for purchasing the MEAP of Math for Programmers I’ve been a math enthusiast my whole life, and only accidentally stumbled into... of AI until she learned some math Fortunately the math chapter of Deep Learning listed a reference on "linear algebra" for students who had never seen the topic before She tracked down this textbook