Columntransformer causes openml to open new setup for otherwise identical pipelines

[hp2500]

Hi there,
i am running experiments where it is important that I can compare the performance of the same setup across different tasks. Generally I have no problems doing this, but whenever I am running a task that has mixed data types in its feature matrix I run into issues.

Specifically, I am using the columntransformer to run different preprocessing steps on categorical and numerical features. I am running the same pipeline on each one of the tasks, but openml creates a new setup for each of these runs, making it impossible to find the runs belonging to each of my pipelines / parameter configurations.

The only thing I can think of that changes from run to run are the columns of the numeric and categorical features, hence the indicator mask that I am passing to the columntransformer is different from run to run. My suspicion is that this is causing openml to treat each pipeline as a different pipeline, creating a new setup for each run, even though the pipeline is technically the same.

Here is a simple example:

import openml
from sklearn.svm import SVC 
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder 

# get task
task = openml.tasks.get_task(3022)

# get dataset object 
data = openml.datasets.get_dataset(task.dataset_id)

# get relevant info from dataset object
X, y, categorical_indicator, attribute_names = data.get_data(dataset_format='array',
                                                            target=data.default_target_attribute)
# make indicator masks
cat = categorical_indicator
num = [not k for k in categorical_indicator]

# make columntransformer
numeric_transformer = make_pipeline(StandardScaler())
categorical_transformer = make_pipeline(OneHotEncoder(handle_unknown='ignore'))
preprocessor = ColumnTransformer(
transformers=[('num', numeric_transformer, num), ('cat', categorical_transformer, cat)])

# make pipeline
clf = SVC(gamma = 'scale', random_state=1)
pipe = make_pipeline(preprocessor, clf)

# run task
run = openml.runs.run_model_on_task(pipe, task, avoid_duplicate_runs=True)
run.publish()

# check setup 
openml.runs.get_run(run.run_id).setup_id

8231820

# get task
task = openml.tasks.get_task(23)

# get dataset object 
data = openml.datasets.get_dataset(task.dataset_id)

# get relevant info from dataset object
X, y, categorical_indicator, attribute_names = data.get_data(dataset_format='array',
                                                            target=data.default_target_attribute)
# make indicator masks
cat = categorical_indicator
num = [not k for k in categorical_indicator]

# make columntransformer
numeric_transformer = make_pipeline(StandardScaler())
categorical_transformer = make_pipeline(OneHotEncoder(handle_unknown='ignore'))
preprocessor = ColumnTransformer(
transformers=[('num', numeric_transformer, num), ('cat', categorical_transformer, cat)])

# make pipeline
clf = SVC(gamma = 'scale', random_state=1)
pipe = make_pipeline(preprocessor, clf)

# run task
run = openml.runs.run_model_on_task(pipe, task, avoid_duplicate_runs=True)
run.publish()

# check setup 
openml.runs.get_run(run.run_id).setup_id

8231821

This is not the case when I am not using the columntransformer (even though I should)

# get task
task = openml.tasks.get_task(3022)

# get dataset object 
data = openml.datasets.get_dataset(task.dataset_id)

# make pipeline
clf = SVC(gamma = 'scale', random_state=1)
pipe = make_pipeline(StandardScaler(), clf)

# run task
run = openml.runs.run_model_on_task(pipe, task, avoid_duplicate_runs=True)
run.publish()

# check setup 
openml.runs.get_run(run.run_id).setup_id

8231791

# get task
task = openml.tasks.get_task(23)

# get dataset object 
data = openml.datasets.get_dataset(task.dataset_id)

# make pipeline
clf = SVC(gamma = 'scale', random_state=1)
pipe = make_pipeline(StandardScaler(), clf)

# run task
run = openml.runs.run_model_on_task(pipe, task, avoid_duplicate_runs=True)
run.publish()

# check setup 
openml.runs.get_run(run.run_id).setup_id

8231791

Is this behavior intended?
Is there a non-hacky solution to this problem?

@amueller

[amueller]

you could use a callable. Looks like get_dataset returns categorical types correctly annotated:

def cont(X):
	return X.dtypes != 'category'

def cat(X):
	return X.dtypes == 'category'

[amueller]

Tried to solve this in #777.
My solution above doesn't work for two reasons: run_model_on_task doesn't use dataframes internally, and callables are not allowed in column transformer (for no reason that I could figure out).

[mfeurer]

Is this behavior intended?

No, since this is a hyperparameter it should not create a new flow. The sklearn extension was created before the column transformer and apparently we missed this when adding support for the column transformer.

run_model_on_task doesn't use dataframes internally

I'm wondering whether the use of dataframes should be the default (and the users can switch back to numpy arrays) or the only option. Can everything in scikit-learn handle dataframes by now? According to the docs (https://scikit-learn.org/stable/developers/contributing.html#fitting) this is not necessary?

and callables are not allowed in column transformer

This was added by @janvanrijn in #523 and I'm not 100% sure why we don't allow callables. My best guess is that we don't have a mechanism to upload and reinstantiate callables yet.

[janvanrijn]

This was added by @janvanrijn in #523 and I'm not 100% sure why we don't allow callables. My best guess is that we don't have a mechanism to upload and reinstantiate callables yet.

I would guess the same, or at least call it untested behavior.

[amueller]

Everything can take a dataframe and I'd suggest making that the default behavior.
That's a more tricky fix, though, see #777.
I probably won't have time to finish that off this week.

@hp2500 do you mean these create different setups or different flows? I thought the issue was that they create different setups, which different hyper parameters should create.

If you use a callable, these would be the same setups, which was the goal here (we want to compare hyper-parameter settings across datasets for meta-learning purposes).

I haven't tested the callable, but #777 has a prototype. @hp2500 if you like you can try #777 but it's quite untested and I have some other stuff to do this week :-/

[mfeurer]

do you mean these create different setups or different flows

My bad, @hp2500 indeed said setups. Still an interesting question how to handle this in the general case (if the user passes a list of values for example). If it is the same callable it should result in the same setup though and this special case would be straight-forward.

[mfeurer]

If you use a callable, these would be the same setups

Serializing a callable should be easily feasible with the inspect package. However, it would be hard to follow for other functions which are called by the callable and are not part of the standard library. Also, how would we make sure that there are no malicious callables in our code? We could ask the user for confirmation prior to executing a callable?

[Neeratyoy]

Under the changes pushed in #777, the following example, modified slightly from the OP, works now on the current develop branch:

import openml
from sklearn.svm import SVC 
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder 


def cont(X):
	return X.dtypes != 'category'

def cat(X):
	return X.dtypes == 'category'


def debug_ct(task_id):
	task = openml.tasks.get_task(task_id)

	# make columntransformer
	preprocessor = ColumnTransformer(
		transformers=[
			('num', StandardScaler(), cont), 
			('cat', OneHotEncoder(handle_unknown='ignore'), cat)
		]
	)

	# make pipeline
	clf = SVC(gamma = 'scale', random_state=1)
	pipe = make_pipeline(preprocessor, clf)

	# run task
	run = openml.runs.run_model_on_task(pipe, task, avoid_duplicate_runs=True)
	run.publish()

	new_run = openml.runs.get_run(run.run_id)
	# check setup 
	print("For Run ID {}, the setup ID: {}".format(run.run_id, new_run.setup_id))


openml.config.start_using_configuration_for_example()

debug_ct(23)
debug_ct(230)

openml.config.stop_using_configuration_for_example()

Internally, run_model_on_task runs by default on dataframe, and hence the functions cat and cont are essential to have ColumnTransformer working.

@mfeurer should this issue be closed or shall we provide some helper code or example to have more generic sklearn pipelines be accessible to the user?

[Neeratyoy]

I believe, this issue could be closed once #982 is merged.

[mfeurer]

Okay, this should finally be done with #982 being merged now!