What are the limitations of the gradient descent algorithm?

Disadvantages of Batch Gradient Descent

• Perform redundant computation for the same training example for large datasets.
• Can be very slow and intractable as large datasets may not fit in the memory.
• As we take the entire dataset for computation we can update the weights of the model for the new data.

What can be the disadvantage if the learning rate is very high in gradient descent?

When the learning rate is too large, gradient descent can inadvertently increase rather than decrease the training error. […] When the learning rate is too small, training is not only slower, but may become permanently stuck with a high training error.

What is the effect of the step size?

 A large step size helps increase finding an initial solutions however the quality of that solution is low.  Adding a minimum step size eliminates the addition of redundant nodes.  Using a node selection/rejection heuristic reduces the number of explored nodes.

What is SGD stochastic gradient descent What’s the difference with the usual gradient descent?

The only difference comes while iterating. In Gradient Descent, we consider all the points in calculating loss and derivative, while in Stochastic gradient descent, we use single point in loss function and its derivative randomly.

Does Stochastic Gradient Descent prevent Overfitting?

Since there are standard generalization bounds for predictors which achieve a large margin over the dataset, we get that asymptotically, gradient descent does not overfit, even if we just run it on the empirical risk function without any explicit regu- larization, and even if the number of iterations T diverges to …

What is the limitation of high learning rate?

If your learning rate is set too low, training will progress very slowly as you are making very tiny updates to the weights in your network. However, if your learning rate is set too high, it can cause undesirable divergent behavior in your loss function.