Grid Search
Definition:
Process:
1. A set of hyperparameters to be tuned is defined, and a range of values for each hyperparameter is specified.
2. A grid is formed by taking the Cartesian product of the hyperparameter values, and for each combination of hyperparameters, a model is trained and evaluated using a cross-validation technique.
3. The performance of the model on the validation set is used as the objective function to be optimized.
4. Finally, the hyperparameter combination that results in the best performance is chosen as the optimal set of hyperparameters.
Cons: Computationally expensive
Random Search
Process:
1. A range of possible values for each hyperparameter is defined.
2. During training, random combinations of hyperparameters are chosen from this range and used to train the model. The process is repeated for a fixed number of iterations or until a satisfactory set of hyperparameters is found.
Pros:
1. Random Search is often preferred over grid search because it allows for a wider exploration of the hyperparameter space and can often find better hyperparameters with fewer trials.
2. It is also computationally less expensive than an exhaustive grid search when the search space is large or the number of hyperparameters is high.
Bayesian Optimization
Process:
1. Defining the search space
2. Define the objective function
3. Building the probabilistic model
4. Selecting the next set of hyperparameters
5. Evaluating the model with the new hyperparameters
6. Updating the probabilistic model
objective function
acquisition function
Pros:
Bayesian optimization is a powerful and efficient technique for hyperparameter tuning, especially when the objective function is expensive to evaluate or when the search space is large and complex. It can often better hyperparameters than other search algorithms with fewer evaluations, resulting in faster and more accurate models.
Genetic Algorithms
Gradient-based Optimization
网友评论