Comparing Solutions from Saddle Points vs. Local Minimums

reno_rita

Can Saddle Points Provide "Better Solutions" to Machine Learning Models Compared to Local Minimums? The solution to a Machine Learning model (i.e. the final model parameters) are selected by trying to optimize the Loss Function associated with that Machine Learning model. The "best" solution (i.e. "best" choice of model parameters) are those associated with the "global minimum" of this Loss Function (i.e. the smallest value of the Loss Function)...

Originally asked by stats_noob on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Your interpretation of the graph is correct, in this simple example weight taken from the saddle point will perform better than the samples taken at the local minima. Said tat, this 1D example is a bit of an outlier compared to what happen in more realistic scenarios. If we keep in mind the second partial derivative test , used to check if a point of a function is a maximum, minimum or saddle point, then it's easy to see that especially in higher dimensions the probability of a point being a saddle one are higher than being a maximum or a minimum one. When analyzing a point we first compute the determinant of the Hessian matrix of our loss function (I'm using the 2 variables formulation for...

Full answer by Edoardo Guerriero: https://ai.stackexchange.com/a/34447 (CC BY-SA 4.0).

Need it done for your business? UphillTech.