What is hyperparameter tuning?

Introduction

Hyperparameter tuning is a technique used to improve the accuracy of models. If you make a model with scikit-learn and do not set parameters, it will be set with appropriate complexity.

What are hyperparameters?

Parameters that are specified before training to determine the training method, speed, and model complexity.

Method (type)

bergstra12a-04.jpg Source: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf

Grid search

It is a method to decide candidates for each parameter and try all combinations of them. Since all are tried, it is not possible to increase the number of parameter candidates.

Random search

It is a method of deciding a candidate for each parameter and repeating a random combination of parameters n times. It may not be possible to search for a better combination of parameters because we do not try all of them.

Parameter combination

import numpy as np

params_list01 = [1, 3, 5, 7]
params_list02 = [1, 2, 3, 4, 5]

#Grid search
grid_search_params = []
for p1 in params_list01:
    for p2 in params_list02:
        grid_search_params.append(p1, p2)
# append():Add an element to the end of the list

#Random search
random_search_params = []
count = 10
for i in range(count):
    p1 = np.random.choice(params_list01)  # random.choice():Get the contents of the array randomly
    p2 = np.random.choice(params_list02)
    random_search_params.append(p1, p2)

scikit-learn

Click here for scikit-learn reference

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
params = {
    "max_depth": [2, 4, 6, 8, None],
    "n_estimators": [50,100,200,300,400,500],
    "max_features": range(1, 11),
    "min_samples_split": range(2, 11),
    "min_samples_leaf": range(1, 11)
}

#Grid search
gscv = GridSearchCV(RandomForestRegressor(), params, cv=3, n_jobs=-1, verbose=1)
gscv.fit(X_train_valid, y_train_valid)
            
print("Best score: {}".format(gscv.best_score_))
print("Best parameters: {}".format(gscv.best_params_))

#Random search
rscv = RandomizedSearchCV(RandomForestRegressor(), params, cv=3, n_iter=10, n_jobs=-1, verbose=1)
rscv.fit(X_train_valid, y_train_valid)
            
print("Best score: {}".format(rscv.best_score_))
print("Best parameters: {}".format(rscv.best_params_))

in conclusion

When asking which one should be adopted, a random search is performed. It seems that a good combination of parameters can be found efficiently.

reference

Book: Data analysis technology that wins with Kaggle (Technical Review)

Recommended Posts

What is hyperparameter tuning?
Hyperparameter tuning
Hyperparameter tuning 2
What is namespace
What is copy.copy ()
What is dotenv?
What is Linux
What is klass?
What is SALOME?
What is Linux?
What is python
What is hyperopt?
What is Linux
What is pyvenv
What is __call__
What is Linux
What is Python
What is a distribution?
What is Piotroski's F-Score?
What is Raspberry Pi?
[Python] What is Pipeline ...
What is Calmar Ratio?
What is a terminal?
[PyTorch Tutorial ①] What is PyTorch?
What is a hacker?
What is JSON? .. [Note]
What is Linux for?
What is a pointer?
What is ensemble learning?
What is TCP / IP?
What is Python's __init__.py?
What is an iterator?
What is UNIT-V Linux?
[Python] What is virtualenv
What is machine learning?
What is Minisum or Minimax?
What is Linux? [Command list]
What is Logistic Regression Analysis?
What is the activation function?
What is an instance variable?
What is a decision tree?
What is a Context Switch?
What is Google Cloud Dataflow?
[DL] What is weight decay?
[Python] Python and security-① What is Python?
What is a super user?
Competitive programming is what (bonus)
[Python] * args ** What is kwrgs?
What is a system call
[Definition] What is a framework?
What is the interface for ...
What is Project Euler 3 Acceleration?
What is a callback function?
What is the Callback function?
What is a python map?
What is your "Tanimoto coefficient"?
Python Basic Course (1 What is Python)
What is Python? What is it used for?
[Python] What is a zip function?
[Python] What is a with statement?
What is Reduced Rank Ridge Regression?