Which optimization algorithm adapts the learning rate for each parameter based on its gradient history?

The optimization algorithm that adapts the learning rate for each parameter based on its gradient history is called "Adaptive Moment Estimation" or "Adam".

Here's a step-by-step explanation of how it works:

1. Initialize the parameters: Adam starts with an initial estimate of the parameters of the model.

2. Compute the gradient: In each iteration, Adam computes the gradient of the loss function with respect to the parameters.

3. Update biased first and second moment estimates: Adam maintains an exponentially decaying average of past gradients and squared gradients.

4. Correct bias in moment estimates: Adam corrects the bias in its estimates to handle the initialization at the origin.

5. Compute parameter update: Adam computes the update of the parameters based on the corrected estimates.

6. Apply the update: Finally, Adam applies the computed update to the parameters.

This process is repeated until the algorithm converges to the optimal parameters. The adaptive learning rate for each parameter makes Adam particularly effective when dealing with sparse gradients or noisy data.