Which algorithm is known for finding optimal solutions by iteratively minimizing a cost function?

The algorithm known for finding optimal solutions by iteratively minimizing a cost function is called Gradient Descent.

Here are the steps it follows:

1. Initialize: Start with random values for the parameters (weights).

2. Compute: Calculate the cost function, which measures the error of the prediction of the model.

3. Gradient Calculation: Compute the gradient of the cost function. The gradient is a derivative operation that calculates the slope of the function at a specific point.

4. Update Parameters: Adjust the parameters in the direction that minimizes the cost function. This is done by subtracting the gradient of the cost function from the current parameters.

5. Iterate: Repeat steps 2-4 until the cost function is minimized to a satisfactory level or after a certain number of iterations.

6. Terminate: The algorithm stops when the cost function is at its minimum. The parameters at this point are considered to be the optimal parameters.