For the case of gradient descent we can use a fixed steplength value (and indeed can compute a conservative Lipschitz value that will always produce descent, see Section appendix), a diminishing steplength scheme, or an adjustable method like backtracking line search. The generic practical considerations associated with each method still exist here (as discussed in Chapters 6 and 7) i.e., with gradient descent we must choose a steplength scheme, and Newton's method is practically limited to cases when $N$ is of moderate value (e.g., in the thousands). The Least Squares cost function for linear regression is always convex regardless of the input dataset, hence we can easily apply either gradient descent or Newton's method in order to minimize it. Because of this we can easily apply either gradient descent or Newton's method in order to minimize it. However the Least Squares cost function for linear regression can mathematically shown to be - in general - a convex function for any dataset (this is because one can show that it is always a convex quadratic - see Section 8.1.6 for details). There we saw the elliptical contours and 'upward bending' shape of the surface indeed confirms the function's convexity in that case. In the previous example we plotted the contour/surface for the Least Squares cost function for linear regression on a specific dataset. Below we provide one such numpy heavy version of the Least Squares implementation shown previously which is far more efficient. This means we do away with the explicit for loop over each of our $P$ points and make the same computations (numerically speaking) for every point simultaneously. In fact Python code, employing heavy use of numpy functions, can often execute almost as fast a raw C implementation itself.īroadly speaking, when scribing a Pythonic function with heavy use of numpy functionality one tries to package each step of computation - which previously was being formed sequentially at each data point - together for the entire dataset simultaneously. numpy is an API for some very efficient vector/matrix manipulation libraries written in C. It is easy to get around most of this inefficiency by replacing explicit for loops with numerically equivalent operations performed using operations from the numpy library. However explicit for loops (including list comprehensions) written in Python are rather due to the very nature of the language (e.g., it being a dynamically typed interpreted language). Second, we can notice that this really is a direct implementation of the algebraic form of the cost in equation (13), where we think of the cost modularly the sum of squared errors of a linear model of input against its corresponding output.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |