evaluate_regressors_by_problem

tsml_eval.evaluation.evaluate_regressors_by_problem(load_path, regressor_names, dataset_names, save_path, resamples=1, load_train_results=False, error_on_missing=True, continue_on_missing=False, eval_name=None, verify_results=True, verbose=False)

Evaluate multiple regressors on multiple datasets from file using standard paths.

Finds files using regressor, dataset and resample names. It is expected the common tsml-eval file structure of {regressor}/Predictions/{dataset}/{split}Resample{resample}.csv is followed.

Writes multiple csv files and figures to save_path, one for each statistic evaluated. Provides a summary csv file with the average statistic and average rank for each regressor.

Parameters:

load_pathstr or list of str

The path to the collection of regressor result files to evaluate. If load_path is a list, it will load results from each path in the list. It is expected that regressor_names and dataset_names are lists of lists with the same length as load_path.

regressor_nameslist of str, list of tuple or list of list

The names of the regressors to evaluate. A length 2 tuple containing strings can be used to specify a regressor name to load from in the first item and a regressor name to use in the evaluation results in the second. If load_path is a list, regressor_names must be a list of lists with the same length as load_path.

dataset_namesstr, list of str or list of list

The names of the datasets to evaluate. If a list of strings, each item is the name of a dataset. If a string, it is the path to a file containing the names of the datasets, one per line. If load_path is a list, dataset_names must be a list of lists with the same length as load_path.

save_pathstr

The path to save the evaluation results to.

resamplesint or list of int, default=1

The resamples to evaluate. If int, evaluates resamples 0 to resamples-1. if None, treats resample as empty i.e. {split}Resample.csv.

load_train_resultsbool, default=False

Whether to load train results as well as test results.

error_on_missingbool, default=True

Whether to raise an error if results are missing.

continue_on_missingbool, default=False

Whether to continue the evaluation if results are missing. If False, removes datasets with missing results from the evaluation. If True, keeps all datasets but does not include summary results, figures or p-values. Treats any missing stat as NaN.

eval_namestr, default=None

The name of the evaluation, used in save_path.

verify_resultsbool, default=True

If the verification should be performed on the loaded results values.

verbosebool, default=False

If verbose output should be printed.