evaluate_clusterers_by_problem

tsml_eval.evaluation.evaluate_clusterers_by_problem(load_path, clusterer_names, dataset_names, save_path, resamples=1, load_test_results=True, error_on_missing=True, continue_on_missing=False, eval_name=None, verify_results=True, verbose=False)

Evaluate multiple clusterers on multiple datasets from file using standard paths.

Finds files using clusterer, dataset and resample names. It is expected the common tsml-eval file structure of {clusterer}/Predictions/{dataset}/{split}Resample{resample}.csv is followed.

Writes multiple csv files and figures to save_path, one for each statistic evaluated. Provides a summary csv file with the average statistic and average rank for each clusterer.

Parameters:

load_pathstr or list of str

The path to the collection of clusterer result files to evaluate. If load_path is a list, it will load results from each path in the list. It is expected that clusterer_names and dataset_names are lists of lists with the same length as load_path.

clusterer_nameslist of str, list of tuple or list of list

The names of the clusterers to evaluate. A length 2 tuple containing strings can be used to specify a clusterer name to load from in the first item and a clusterer name to use in the evaluation results in the second. If load_path is a list, clusterer_names must be a list of lists with the same length as load_path.

dataset_namesstr, list of str or list of list

The names of the datasets to evaluate. If a list of strings, each item is the name of a dataset. If a string, it is the path to a file containing the names of the datasets, one per line. If load_path is a list, dataset_names must be a list of lists with the same length as load_path.

save_pathstr

The path to save the evaluation results to.

resamplesint or list of int, default=1

The resamples to evaluate. If int, evaluates resamples 0 to resamples-1. if None, treats resample as empty i.e. {split}Resample.csv.

load_test_resultsbool, default=True

Whether to load test results as well as train results.

error_on_missingbool, default=True

Whether to raise an error if results are missing.

continue_on_missingbool, default=False

Whether to continue the evaluation if results are missing. If False, removes datasets with missing results from the evaluation. If True, keeps all datasets but does not include summary results, figures or p-values. Treats any missing stat as NaN.

eval_namestr, default=None

The name of the evaluation, used in save_path.

verify_resultsbool, default=True

If the verification should be performed on the loaded results values.

verbosebool, default=False

If verbose output should be printed.