Incompatible with Google Colaboratory

[PositronBeam]

Incompatible with Google Colaboratory

[calz1]

This would be really neat to have working. As Google Colab gives free GPUs, this would let a lot more people try out Autokeras...

[vincent-hui]

I encounter the same issue.

[haifeng-jin]

@PositronBeam Thank you for your report.
We haven't tested with Google Collaboratory yet.
Will come back to this soon.

Thanks.

[vincent-hui]

Hi @jhfjhfj1 , would you mind telling us whether you tested with Google Collaboratory?

Thank a lot

[haifeng-jin]

@vincent-hui , Thank you for your follow up.
We have changed the multiprocessing into torch.multiprocessing.
But we haven't tested with the Google Colaboratory yet.

[vincent-hui]

@jhfjhfj1 I just tried autokeras-0.2.7 with the Google Colaboratory
Google Colaboratory runtime dies when I run

from keras.datasets import mnist
from autokeras import ImageClassifier

if __name__ == '__main__':
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train = x_train.reshape(x_train.shape + (1,))
    x_test = x_test.reshape(x_test.shape + (1,))

    clf = ImageClassifier(verbose=True, augment=False)
    clf.fit(x_train, y_train, time_limit=12 * 60 * 60)
    clf.final_fit(x_train, y_train, x_test, y_test, retrain=True)
    y = clf.evaluate(x_test, y_test)
    print(y * 100)

Here is the output message

Using TensorFlow backend.

Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz
11493376/11490434 [==============================] - 2s 0us/step
Initializing search.
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43]
Initialization finished.
Training model  0

[vincent-hui]

@jhfjhfj1 I just tried autokeras-0.2.11 with the Google Colaboratory
Google Colaboratory runtime dies when I use autokeras-0.2.11
Here is the output message

Using TensorFlow backend.
/usr/local/lib/python3.6/dist-packages/tqdm/autonotebook/__init__.py:14: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
  " (e.g. in jupyter console)", TqdmExperimentalWarning)


Initializing search.
Initialization finished.


╒==============================================╕
|               Training model 0               |
╘==============================================╛

[vincent-hui]

I tried autokeras-0.2.13 with the Google Colaboratory. Google Colaboratory runtime dies when running autokeras.

[haifeng-jin]

Help needed.
Does anyone know why it cannot work with Google Collaboratory and how to fix it?
Thanks

[vincent-hui]

@jhfjhfj1
Sorry for my stupid question. In Additional context of the first post of @@PositronBeam , is it a workaround for running autokeras on Google Collaboratory?

[compassNS]

Add google colab check to @PositronBeam

search.py line175 - 180

BEFORE==============================================================

        mp.set_start_method('spawn', force=True)
        pool = mp.Pool(1)
        try:
            train_results = pool.map_async(train, [(graph, train_data, test_data, self.trainer_args,
                                                    		os.path.join(self.path, str(model_id) + '.png'),
self.metric, self.loss, self.verbose)])

AFTER===============================================================

        import sys
        if "google.colab" in sys.modules:
          gcolab = True
        else:
          gcolab = False
          mp.set_start_method('spawn', force=True)
          pool = mp.Pool(1)

        try:
           if gcolab:
              train_results = train((graph, train_data, test_data, self.trainer_args,
                                              os.path.join(self.path, str(model_id) + '.png'),
                                              self.metric, self.loss, self.verbose))
           else:
              train_results = pool.map_async(train, [(graph, train_data, test_data, self.trainer_args,
                                                    	          os.path.join(self.path, str(model_id) + '.png'),
                                                                          self.metric, self.loss, self.verbose)])

[vincent-hui]

Hi @compassNS ,

Would you mind telling me how to run autokeras with Google Colaboratory ?

Thank a lot.

[Frozen-In-Ice]

I tried making the change @compassNS suggested but I got this error:
local variable 'pool' referenced before assignment

[tl-yang]

@chencc6566193 Since this issue is kind of urgent, @jhfjhfj1 reassigned it to me as well. Please feel free to let me know if you had any progress, then we two can co-work on this one. Thanks!

[kafkapple]

Help needed.
Does anyone know why it cannot work with Google Collaboratory and how to fix it?
Thanks

@jhfjhfj1 In my case, it seems that lack of memory was the problem when I use google Colab.
So I think that, some kind of batch generator such as fit_generator something could be useful to handle with the problem.
Do you have any plan to implement this idea?

By the way, autokeras is really easy, and powerful for me to do NAS for image classification!
Thank you so much for your kindness and effort for this :)

Now I'm gonna also try to use autokeras for other data types, such as time-series neuronal / behavior data - convert them into kind of 2-d array or text data.
I saw that module for audio data is also being considered, and it will be really nice for them too!

[tl-yang]

Support GPU on Google Colab

###Solution

• Disable multiprocessing

Reason

• Google Colab doesn't support multiprocessing "spwan" start method. However, cuda only support "spawn" start method. Thus, to enable GPU accelaration on Google Colab, we must run autokeras in sigle process.

[kalyanramu]

Sorry, I am confused about the final resolution. Is it changing the Google Colab Notes to single process or was there a change in the source code. Also, can you please post steps to disable multi-processing?