pickle for SVHN bigger than arff file

[amueller]

For dataset 41081, I get:

1.1G	dataset.arff
2.3G	dataset.pkl.py3

Maybe we should be using joblib instead? My hard drive is getting full because I'm trying to run CC-18 stuff, which is a bit annoying.

[amueller]

potentially related: loading the dataset takes like 10GB of ram which makes running benchmarks on CC-18 more tricky.

[joaquinvanschoren]

+1 for using joblib. Although actually Arrow/Feather seems even better, and will definitely load large datasets a lot faster once they are cached.

We also experienced cases where the file won't unpickle correctly (see #780).
The 10GB RAM usage is because of the ARFF parser, right? Or are you also using so much when you load the pickle?

[amueller]

Could be the arff parser

[amueller]

Ok so here's some numbers for CIFAR10:

arff: 0.63 G

so the first problem is that the class (which is an integer in the original) is given a dtype of category, which is then stored as an object dtype, making the dataframe harder to store.
If we don't do that, we can get substantial gains from joblib compression and go to half the size of the arff. But the real deal is storing it as uint8 as it was original, which renders compression useless and makes it really fast and small.

[amueller]

so looks like this was broken in #548

[amueller]

actually the dtype object for the classes is not the issue, we can probably get all the gain by just storing the data as uint8.

[amueller]

Using joblib compress=3 takes about 3x as long for storing than the standard dump, not sure if that's worth it. For MNIST (which is very redundant), file size goes from 53M (uint8) to 30M (compressed uint8). Current develop branch is 140M.
Loading is only a little slower with joblib than with pickle.

[mfeurer]

potentially related: loading the dataset takes like 10GB of ram

This could be fixed by using a generator to load the data as done for scikit-learn.

Using joblib compress=3 takes about 3x as long for storing than the standard dump, not sure if that's worth it.

I think so, the number of datasets on OpenML will only grow.

[mfeurer]

Hey @amueller we just merged storing image data as uint8. Do you think it's worth pursuing compressing the files? In my opinion it is more important to load fast than to save space, but others might have a different opinion on that.