The idea of feeding the config file with a python file instead of yaml

Input config in yml file

I think that a yaml file is used to input config information when learning and inferring a machine learning model.

However, personally, the fact that variables cannot be used in yaml is a pain. There are variables like anchors and aliases, but since they hold values for each line, they cannot hold partial commonalities, for example, when the parent directories are common.

For example

image: "/path/to/dataset_name/data_version/Images"
label: "/path/to/dataset_name/data_version/Csv"

To

shared_dir = "/path/to/dataset_name/data_version/"

image: shared_dir + "Images"
label: shared_dir + "Csv"

Because I can't write like this, I find it difficult to edit and reuse.

At that time, some people may create only the default key and val placeholders, and then dynamically create the required yml file with the template generator and load it. It's [^ 1]. For me, it would be a hassle to add one processing process.

[^ 1]: I don't know the yaml example, but I've seen it in caffe's prototxt. The background is that the solver (corresponding to optimizer) class of caffe's python API doesn't have a setter method, so I think it was because the parameters had to be given via a file.

Input config in python file

So I came up with the idea that I should take the method of directly eating the python file. Eating here is not from config import cfg, but

Config file written in python

config.py

# config.py

path_root = '/path/to/'

cfg = {
    'key1': path_root + 'val1',
    'key2':path_root + 'val2',
    'some_number': 1+ 3,
}

,

import importlib

cfg_filepath = './config.py'
hoge = importlib.machinery.SourceFileLoader('hoge_module_name', cfg_filepath).load_module()
cfg = hoge.cfg

print(cfg)  # {'key1': '/path/to/val1', 'key2': '/path/to/val2', 'some_number': 4}

It refers to reading the contents of the config by passing it to the loader with the file path. In this case, you can switch the config file you want to read with the argument without changing the load destination of config with the import statement each time. Also, you can handle expression evaluation and objects handled by python. Of course, there is a drawback that it can not be used except for python, but variables make the config file look neat.

Can you really use it?

I wrote it down, but I felt that people wouldn't normally edit the parameters in the config file and experiment, whether it's a yaml file or a python file. If you are going through the command line, you can change only the parameters you want to change with the arguments, or you can change it through the GUI of the AI platform, or you are doing the experiment with jupyter and collecting the parameters in a cell (spitting yaml at runtime as a log). It seems that most of them are, and I have the impression that there is almost no opportunity to directly modify the config file.

As for whether the python file method can be used as in this article, the honest point is, "Although it can be used, it is better to think of another way of giving." If there is a useful case with the method in this article, I will add it later.

Thank you for reading.

References

-Python: Dynamically load the module --CUBE SUGAR CONTAINER