evaluate_classifiers_by_problem

tsml_eval.evaluation.evaluate_classifiers_by_problem(load_path, classifier_names, dataset_names, save_path, resamples=None, load_train_results=False, error_on_missing=True, eval_name=None, verify_results=True, verbose=False)

Evaluate multiple classifiers on multiple datasets from file using standard paths.

Finds files using classifier, dataset and resample names. It is expected the common tsml-eval file structure of {classifier}/Predictions/{dataset}/{split}Resample{resample}.csv is followed.

Writes multiple csv files and figures to save_path, one for each statistic evaluated. Provides a summary csv file with the average statistic and average rank for each classifier.

Parameters:

load_pathstr or list of str

The path to the collection of classifier result files to evaluate. If load_path is a list, it will load results from each path in the list. It is expected that classifier_names and dataset_names are lists of lists with the same length as load_path.

classifier_nameslist of str, list of tuple or list of list

The names of the classifiers to evaluate. A length 2 tuple containing strings can be used to specify a classifier name to load from in the first item and a classifier name to use in the evaluation results in the second. If load_path is a list, classifier_names must be a list of lists with the same length as load_path.

dataset_namesstr, list of str or list of list

The names of the datasets to evaluate. If a list of strings, each item is the name of a dataset. If a string, it is the path to a file containing the names of the datasets, one per line. If load_path is a list, dataset_names must be a list of lists with the same length as load_path.

save_pathstr

The path to save the evaluation results to.

resamplesint or list of int, default=None

The resamples to evaluate. If int, evaluates resamples 0 to resamples-1.

load_train_resultsbool, default=False

Whether to load train results as well as test results.

error_on_missingbool, default=True

Whether to raise an error if results are missing.

eval_namestr, default=None

The name of the evaluation, used in save_path.

verify_resultsbool, default=True

If the verification should be performed on the loaded results values.

verbosebool, default=False

If verbose output should be printed.