Mark a parameter for pytest parametrize that should be skipped ..code:: python @parametrize("number", [2, 3, skip(5, "some reason")])

Return the following statistics of the training result: - dataset name - metric used - best validation score - number

Used in some cases to enforce the only valid model is the dummy model

Makes sure ensemble selection fit method creates an ensemble correctly

Expects ------- * Should only load X_train if it's required * TODO should only load X_test if it's required

Test that classifiers can be added Expects ------- * There should be 0 components initially * There should be 1 compo

Test that preprocessors can be added Expects ------- * There should be 0 components initially * There should be 1 com

Expects ------- * There should be at least one ensemble generated

Expects ------- * There should be at least one models saved * Each model saved should have predictions for the ensemble

Parameters ---------- automl : AutoML An unfitted automl instance attr: str The attribute to test argnames: Uni

This test makes sure that anneal dataset can be fitted and scored. This dataset is quite complex, with NaN, categorical and numerical columns

Currently this method only tests that the methods of AutoSklearnClassifier is able to fit using fit(), fit_ensemble() and refit()

Expects ------- * Should be able to read runs from the backends rundir where runs are tagged {seed}_{numrun}_{budget}

Expects ------- * Should ignore dirs that aren't in format

Expects ------- * Attempting to load the same predictions again will cause the result to be cached * Unloading the cache will cause i

This test makes sure that numpy and pandas produce the same metafeatures. This also is an excuse to fully test anneal dataset, and make sure

Expects ------- * Multiobjective ensembles should return >= 1, #TODO should test it's pareto optimal

Expects ------- * Non-multiobjective instances should have a pareto set of size 1

Expects ------- * For a run to be considered a candidate, it must be better than the best dummy for any one of the objectives.

Expects ------- * Should filter out runs worse than dummy

Expects ------- * Should filter out runs with no loss

Expects ------- * Should have nothing in common between candidates and discarded * Should filter out runs with no "ensemble" predicti

Expects ------- * Should select nbest percentage of candidates * They should be ordered by loss

Expects ------- * Should only select the nbest candidates * They should be ordered by loss

Expects ------- * Should have nothing in common between candidates and discarded * Should not filter out any viable runs if no filter

Expects ------- * If no run is better than a dummy run, the candidates will then consist of the dummy runs.

Test that the feat_types arg is passed to the OneHotEncoder Expects ------- * Construction of SimpleClassificationPipeline to

Check the asklearn2 has the same args as asklearn1 Expects ------- * The set of arguments for AutoSklearnClassifier is the same as

Parameters ---------- dataspec : Dict The spec to make data of expected_shape : Tuple[int, ...] The expected shape o

Test non-seeded random sets of configurations with default dataset properties Expects ------- * All configurations should fit

Tests a non-seeded random set of configurations with sparse, mixed data Loads specific data from <here>/components/data_preprocessing/dataset

Tests a non-seeded random set of configurations with signed data Expects ------- * All configurations should fit, predict and