Backprop
?Download as PPTX, PDF?
0 likes?99 views
Backprop is an algorithm to calculate the derivatives of variables in equations, especially useful for complicated tensor equations like those in neural networks. The document describes a 3-part video series on understanding compute graphs and applying backprop to compute gradients for simple and more complicated equations using Pytorch, with the objective being able to apply backprop to any equation to compute its gradients.
Backprop
- 1. Backprop Avilay Parekh (Machine Learning Engineer) @avilay
- 2. What is backprop? It is an algorithm to calculate the derivatives of variables in equations. Given: ? = ? ? It outputs: ?? ?? Most useful for complicated tensor equations like the ones we see in Neural Nets.
- 3. 3-Part Video Mini-Series 1. Understand Compute Graphs 2. Backprop of a simple equation with Pytorch 3. Backprop of a more complicated equation with Pytorch Objective: You are able apply the backprop algorithm to any equation to compute its gradients.
- 4. Compute Graphs Different ways to draw the compute graphs for the following equations: ? = ? + 2 ? = (? + 2)2 ? = 3(? + 2)2 ? = 5?2 1 + ? ?
- 8. Backprop Different ways to compute gradients for the following equations: ? = 3(? + 2)2 ? = 5?2 1 + ? ?
Editor's Notes
- Welcome! My name is Avilay. I am a Machine Learning engineering based out of Seattle. You can follow me on Twitter @avilay. In this video mini-series we will cover the back prop algorithm.
- So what is Backprop anyway? It is an algorithm used to calculate derivatives. Put simply, given an equation y equals f of x, backprop gives us the derivative of y w.r.t x. Now, of course if y depends on multiple variables we can calculate the partial derivative of y w.r.t x and so on, but you get the idea. Most of us learnt things like the chain rule and the product rule for calculating derivatives in high school. But if we were to try and apply it to really really complicated equations involving multi dimensional tensors it can be pretty tedious. Not to mention hard to get right. But these are the kind of equations we see in Neural Nets. And this is where Backprop shines.
- In this 3 part series the first video will cover how to create compute graphs. We will go through the exercise of creating compute graphs of various equations by hand. In the second part, we will work through the Backprop algo to calculate the gradients of a simple equation. We will also see how the Pytorch APIs make it very easy for us to do this. In the third video we¡¯ll do the same thing, but with a more complicated equation. After these three videos, you should be able to decompose any equation into its compute graph and apply the backprop algo to compute its gradients. See you in the next video!