Custom OpenMLFlow

[PGijsbers]

I want to register a flow referring to the setup we used to conduct the automl benchmark presented at the ICML workshop this year. We want to do this to more easily share results through OpenML. I want to run this by you all.

Now, I know that creating your own flow in openml-python is discouraged ('Flows should not be generated manually'), but for this scenario I don't really see a better way? Other than using e.g. openml-r.

I want to describe that we used the code at our Github repo, and a specific tag, and provide a url to the download for the code. I figured to tag the used AutoML tool (in this case auto-sklearn).
There is no direct way to instantiate the script with the right settings currently, but flow below should help/be enough to configure a reproducible run. Again, the flow is created mainly to link and share results, and provide the best possible pointers for reproducing the flow.

Sketching out how I think the flow should look like:

auto_sklearn_flow = openml.flows.get_flow(15275)  # auto-sklearn 0.5.1
amlb_flow = openml.flows.OpenMLFlow(
	name='automlbenchmark_autosklearn',
	description='Auto-sklearn as set up by the AutoML Benchmark',
	external_version='amlb==0.9',
	parameters=OrderedDict(
		time='240',
		memory='32',
		cores='8'
	),
	parameters_meta_info=OrderedDict(
		time=dict(data_type='int', description='time in minutes'),
		memory=dict(data_type='int', description='memory in gigabytes'),
		cores=dict(data_type='int', description='number of available cores')
	),
	language='English',
	components=OrderedDict(automl_tool=auto_sklearn_flow),	
)

I would still need to find a place for the following information:

source: https://github.com/openml/automlbenchmark/releases/tag/v0.9
subfolder of particular interest: /frameworks/autosklearn

I do see url fields in OpenMLFlow.__init__ but they are set by the server.

I think there are a couple of questions here:

• Do you think I should even try doing this through the Python API?

And if so,

• Should I create only one flow for the entire benchmark, and take the 'automl tool' as a parameter? I think it's more generic, but might make the link to the automl tool's flow harder to find.
• Where to include the code-url.
• Any other things I need to know?

[mfeurer]

Do you think I should even try doing this through the Python API?

In my opinion yes, it should be flexible enough to allow this to an experienced user. We should update the docs based on this experience.

Should I create only one flow for the entire benchmark, and take the 'automl tool' as a parameter? I think it's more generic, but might make the link to the automl tool's flow harder to find.

I'd find that a bit weird.

Where to include the code-url.

Maybe that would be a good addition to the OpenML schema?

[PGijsbers]

In my opinion yes, it should be flexible enough to allow this to an experienced user.

Good to hear! I agree. Due to the delay I went ahead and tried some of this already last week anyway (I figured I could delete flows if needed).

An example of a flow I created here. I opted to create a new flow for each tool used in the benchmark as they are set up in different ways (hopefully matching your intuition of doing all in one flow being 'a bit weird'). I did however include the automl frameworks as subflow wherever we used a Python tool. The script used to create the flows is here.

Maybe that would be a good addition to the OpenML schema?

I think so, it would make it much clearer/easier to share flows that represent custom scripts.

---

I also uploaded some run results, but that was a little bit more difficult (required two minor changes).
It might be due to me trying to provide the predictions through data_content instead of an arff-file.
I can look into that more, and/or try if the problems also persist if I try to share arff-files directly instead.

[PGijsbers]

Post for later reference.
When uploading runs for said flows, I have to circumvent the following checks:

• run.publish requires the flow to have a model. In my case I don't have the model anymore, just the predictions and parameter settings. Parameter settings were already provided through the OpenMLRun constructor. I think this should be sufficient.
• run.publish assumes the flow to have an extension, when dumping the arff file. I don't think this is a good assumption, users should be able to upload flows of frameworks which do not (yet) have an extension.

Commenting out both publishes the run just fine, example.

[mfeurer]

Added via #932.