evaluate_classifiers_by_problem

tsml_eval.evaluation.evaluate_classifiers_by_problem(load_path, classifier_names, dataset_names, save_path, resamples=1, load_train_results=False, error_on_missing=True, continue_on_missing=False, eval_name=None, verify_results=True, verbose=False)

Evaluate multiple classifiers on multiple datasets from file using standard paths.

Finds files using classifier, dataset and resample names. It is expected the common tsml-eval file structure of {classifier}/Predictions/{dataset}/{split}Resample{resample}.csv is followed.

Writes multiple csv files and figures to save_path, one for each statistic evaluated. Provides a summary csv file with the average statistic and average rank for each classifier.

Parameters:

load_pathstr or list of str

The path to the collection of classifier result files to evaluate. If load_path is a list, it will load results from each path in the list. It is expected that classifier_names and dataset_names are lists of lists with the same length as load_path.

classifier_nameslist of str, list of tuple or list of list

The names of the classifiers to evaluate. A length 2 tuple containing strings can be used to specify a classifier name to load from in the first item and a classifier name to use in the evaluation results in the second. If load_path is a list, classifier_names must be a list of lists with the same length as load_path.

dataset_namesstr, list of str or list of list

The names of the datasets to evaluate. If a list of strings, each item is the name of a dataset. If a string, it is the path to a file containing the names of the datasets, one per line. If load_path is a list, dataset_names must be a list of lists with the same length as load_path.

save_pathstr

The path to save the evaluation results to.

resamplesint or list of int, default=1

The resamples to evaluate. If int, evaluates resamples 0 to resamples-1. if None, treats resample as empty i.e. {split}Resample.csv.

load_train_resultsbool, default=False

Whether to load train results as well as test results.

error_on_missingbool, default=True

Whether to raise an error if results are missing.

continue_on_missingbool, default=False

Whether to continue the evaluation if results are missing. If False, removes datasets with missing results from the evaluation. If True, keeps all datasets but does not include summary results, figures or p-values. Treats any missing stat as NaN.

eval_namestr, default=None

The name of the evaluation, used in save_path.

verify_resultsbool, default=True

If the verification should be performed on the loaded results values.

verbosebool, default=False

If verbose output should be printed.