Can SGD get stuck?

Stochastic gradient descent (SGD) is widely used in machine learning. More specifically, SGD will not get stuck at “sharp” local minima with small diameters, as long as the neighborhoods of these regions contain enough gradient information. The neighborhood size is controlled by step size and gradient noise.

Can gradient descent stuck at saddle point?

When the gradient \nabla f(x) is equal to \vec{0}, the point is called a critical point, and gradient descent algorithm will get stuck. For (strongly) convex functions, there is a unique critical point that is also the global minimum. y = x_1^2 – x_2^2. Such algorithms may get stuck at saddle points.

How can we avoid stuck in local minimum?

Momentum, simply put, adds a fraction of the past weight update to the current weight update. This helps prevent the model from getting stuck in local minima, as even if the current gradient is 0, the past one most likely was not, so it will as easily get stuck.

Why are saddle points problematic?

A typical problem for both local minima and saddle-points is that they are often surrounded by plateaus of small curvature in the error. While gradient descent dynamics are repelled away from a saddle point to lower error by following directions of negative curvature, this repulsion can occur slowly due to the plateau.

Is saddle point global minimum?

Saddle points, unlike local minima, are easily escapable.” Local minima exist, but are very close to global minima in terms of objective functions, and theoretical results suggest that some large functions have their probability concentrated between the index (the critical points) and the objective function.

Does Momentum avoid local minima?

A small value of momentum cannot reliably avoid local minima, and can also slow down the training of the system. Momentum also helps in smoothing out the variations, if the gradient keeps changing direction. A right value of momentum can be either learned by hit and trial or through cross-validation.

Why saddle point is important?

Saddle points represent the highest energy points that must be traversed for the transformation of one configuration to another.