tsml-eval Results Evaluation

This notebook covers the evaluation module of tsml-eval, the primary focus of which is to aid the comparison of multiple machine learning algorithms with each other. While in this example we use the classification learning task, there are functions for clustering, regression and forecasting also available.

​

Comparing Multiple Estimators on Multiple Datasets

In this example, we show a simple example for using the evaluation module to compare multiple classifiers on multiple datasets. For mor information on configuring the functions, see the function documentation.

There are alternate methods of inputting results available. evaluate_classifiers takes a list of ClassifierResults objects, and evaluate_classifiers_from_file takes a list of filepaths to the result files.

import pickle

import matplotlib.pyplot as plt

from tsml_eval.evaluation import evaluate_classifiers_by_problem

classifiers = ["ROCKET", "TSF", "1NN-DTW"]
datasets = ["Chinatown", "ItalyPowerDemand", "Trace"]

evaluate_classifiers_by_problem(
    "../tsml_eval/testing/_test_results_files/classification/",
    classifiers,
    datasets,
    "./generated_results/",
    resamples=1,
    eval_name="ExampleEval",
)

All results present, continuing evaluation.

Estimators (3): ['1NN-DTW', 'ROCKET', 'TSF']

Datasets (3): ['Chinatown', 'ItalyPowerDemand', 'Trace']

Resamples (1): [0]

The function will write various .csv files and figures to the chosen directory, displaying and summarising the results of the evaluation.

with open("./generated_results/ExampleEval/ExampleEval_summary.csv", "rb") as f:
    for _ in range(3):
        print(f.readline().decode("utf-8").strip())

testAccuracy,ROCKET,TSF,1NN-DTW
testAccuracyMean,0.9831551668286362,0.979591836734694,0.9747327502429544
testAccuracyAvgRank,1.3333333333333333,2.0,2.6666666666666665

with open(
    "./generated_results/ExampleEval/Accuracy/figures/"
    "ExampleEval_accuracy_critical_difference.pickle",
    "rb",
) as f:
    fig = pickle.load(f)
plt.show()

with open(
    "./generated_results/ExampleEval/Accuracy/figures/scatters/"
    "ROCKET/ExampleEval_accuracy_scatter_ROCKET_TSF.pickle",
    "rb",
) as f:
    fig = pickle.load(f)
plt.show()

---

Generated using nbsphinx. The Jupyter notebook can be found here.